qualitative research methods for evaluation

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical Literature
• Classical Reception
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Archaeology
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Agriculture
• History of Education
• History of Emotions
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Variation
• Language Families
• Language Evolution
• Language Reference
• Language Acquisition
• Lexicography
• Linguistic Theories
• Linguistic Typology
• Linguistic Anthropology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Modernism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Culture
• Music and Media
• Music and Religion
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Society
• Law and Politics
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Oncology
• Medical Toxicology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Medical Ethics
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Games
• Computer Security
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Neuroscience
• Cognitive Psychology
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business History
• Business Ethics
• Business Strategy
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Methodology
• Economic History
• Economic Systems
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Theory
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Politics and Law
• Public Policy
• Public Administration
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

A newer edition of this book is available.

• < Previous chapter
• Next chapter >

32 Evaluating Qualitative Research

Jeasik Cho, Department of Educational Studies, University of Wyoming

Allen Trent, College of Education, University of Wyoming

• Published: 04 August 2014
• Cite Icon Cite
• Permissions Icon Permissions

This chapter addresses a wide range of theories and practices related to the evaluation of qualitative research (EQR). First, six categories of EQR are presented: (1) a positivist category, (2) Lincoln and Guba’s alternative category, (3) a “subtle-realist” category developed by Hammersley and Atkinson, and Seale, (4) a general EQR category, (5) a category of post-criteriology, and (6) a post-validity category. Second, evaluation strategies for EQR are offered by providing a variety of actual examples. Third, the chapter discusses a path forward for EQR that includes both internal and external elements. The chapter concludes with a holistic view of EQR needed to collectively construct/confront inner and outer challenges to qualitative paradigms in the twenty-first century. Twenty-first-century criteria supported include thought-provoking ideas, innovative methodology, performative writing, and global ethics and justice mindedness.

Quality is elusive, hard to specify, but we often feel we know it when we see it. In this respect research is like art rather than science. – Seale, 2002 , p. 102

Criteria in the 21st century are not one-dimensional. – Lichtman, 2006 , p. 197

We feel exactly the same way that frontier scholars of grounded theory Juliet Corbin and Anselm Strauss (2008) feel regarding the evaluation of qualitative research:

I feel paralyzed, unsure of where to begin, or what to write. As I search the literature, I find that evaluation is necessary but there is little consensus about what that evaluation should consist of. Are we judging for “validity” or would it be better to use terms like “rigor”...“trustworthiness,”... or “goodness,”... or something called “integrity”... when referring to qualitative evaluation? (p. 297)

Let us select the term validity . “Validity has been referred to many ways, including successor validity, catalytic validity, interrogated validity, transgressive validity, imperial validity, simulacra/ironic validity, situated validity, and voluptuous validity” ( Altheide & Johnson, 2011 , pp. 584–585), and a review of the qualitative literature tells us that there are many more definitions.

Why is it so hard to get started “evaluating” qualitative research? Patton (2002) notes that “some of the confusion that people have in assessing qualitative research stems from thinking it represents a uniform perspective, especially in contrast to quantitative research. This makes it hard for them to make sense of the competing approaches within qualitative inquiry” (p. 543).

So, to evaluate qualitative research, shall we simply follow Altheide and Johnson’s (2011) lead? In their chapter in the Sage Handbook of Qualitative Research , entitled “Reflections on Interpretive Adequacy in Qualitative Research,” their approach was threefold: they updated their well-known article “Criteria for Assessing Interpretive Validity in Qualitative Research” ( Altheide & Johnson, 1994 ), they called their ideas about this job “analytical realism,” and they proposed an “evidentiary narrative” embedded in “a symbolic interactionist perspective” (p. 582) that goes against neo-positivist, scientific, or evidence-based research. We are impressed with their deep philosophical, provocative ideas on developing a new grand quality criterion in response to the current scientific, evidence-based movement that devalues an ideal of qualitative research, but we are more interested in exploring a broader sense of evaluative criteria in qualitative research. We call our approach evaluating qualitative research (EQR).

By and large, we are baffled by at least three issues regarding the evaluation of qualitative research: little agreement with the nature of evaluation in qualitative research, a continuous impact of traditional positivist evaluation criteria on qualitative research, and a broad political discourse on the politics of evidence. At least, however, we agree with Schwandt’s (2002) viewpoint that constructing an evaluation lens that involves general and specific accounts of what we might hope to find in a good study is exciting intellectual work. Schwandt’s four general approaches to evaluating qualitative research are to use (1) universal conventional criteria, (2) alternative criteria of trustworthiness and authenticity, (3) pragmatic criteria, and (4) subtle realist criteria of validity and relevance. Although we are impressed with his scheme for a developmental perspective on EQR, our feeling is that this kind of framework is, by itself, something like recreating what has already been deemed disagreeable in this field.

Despite the field’s confusion, disagreements, and our perplexed reaction, our thesis on EQR in this chapter is clear. We express a very simple but meaningful perspective on the evaluation of the processes and products of qualitative research. Our perspective is threefold. First, because we observe that EQR is seen as a relatively cohesive discourse (e.g., a huge number of journal articles and book chapters start with Lincoln and Guba’s [1985] seminal book, Naturalistic Inquiry, and an equally large number of qualitative studies reference and rely on Lincoln and Guba’s [1985] construction of trustworthiness criteria), we want to provide “a sketch of EQR” to categorically describe qualitative differences among many different theoretical and practical ideas related to qualitative research evaluation. Second, we provide several evaluation strategies for EQR. And third, we discuss a path forward for EQR that includes both internal and external elements. We conclude this chapter with a beehive metaphor, which gives a holistic view of the kind of EQR needed to collectively construct, collaborate, and confront inner and outer challenges to qualitative paradigms in the twenty-first century.

Evaluation of Qualitative Research: Six Categories

Under the umbrella of qualitative research over the past three decades, the EQR subfield of study has gradually developed in breadth and depth, along with the blossoming of qualitative inquiry adopted in almost all fields of social science. Relatively speaking, EQR is seen as cohesive because Lincoln and Guba’s (1985) discourse on trustworthiness criteria has been accepted as the platform for EQR. Even though these trustworthiness criteria are still considered essential in discussing the quality of qualitative research, different discourses are available.

As talk of paradigm has broadened, the platform for EQR has changed as well ( Guba & Lincoln, 1989 ). Those who operate from post-modern and post-structuralist traditions criticize trustworthiness criteria as another version of traditional or foundational approaches ( Scheurich, 1996 ). Defining validity is another issue. Some use the terms “criteria” and “validity” interchangeably, drawing on a philosophical and/or evaluation discourse ( Creswell, 2006 ; Schwandt, 2002 ; Seale, 1999 ). Others use validity as a broad epistemological concept to justify an ideal of qualitative inquiry ( Lather, 1986 ). As mentioned earlier, such terms as validity, rigor, trustworthiness, goodness, integrity, and so on are interpreted in many different ways by many different people. Lichtman’s (2006 , 2009) position on EQR provides a good explanation:

At this point, I caution you to be careful as you review criteria for judging qualitative research. Several viewpoints are in play. One group contends that we need to return to research that is more scientific, but I believe that is not necessarily the majority viewpoints. Others see the field as still in a state of flux... The climate of the world of educational research is such that there is increased accountability and standardization and control. The field has become more politicized than it once was.... It is not possible, nor is it desirable, to reach any kind of consensus about what standards should be adopted... the field is not unified... reviewers of journals often embrace a kind of generic criteria. Although they review articles in the health field, the points they make are applicable to education....[Although] the issue of judging, quality, and rigor is very much alive... it is clear that the issue of quality is not yet resolved. (2006, pp. 231–232)

Considering the field’s disparity, as well as the seeming urgent need for some sort of resolution, our sketch of EQR is categorical in pointing out qualitative differences among many different theoretical and practical ideas. We present six categories of EQR: (1) a positivist category; (2) Lincoln and Guba’s alternative category; (3) a “subtle-realist” category developed by Hammersley and Atkinson, and Seale; (4) a general EQR category; (5) a category of post-criteriology; and (6) a post-validity category.

We hope these categories are a useful and meaningful way of sketching a broad view of EQR. We see the six categories as a map that one can use to start making sense of EQR. This sketched map is our own, and others may see the field of EQR differently. We interpret the field of EQR as evolving at present because choosing a set of evaluative criteria in and of itself is socially constructed and politically driven in nature. Therefore, these six categories should not be interpreted as either hierarchical or linear. Simply put, each is a distinctly different category relying on its own specific criteria ( Cho & Trent, 2006 ; Tracy, 2010 ). We would like our six categories to be seen as providing a holistic perspective, one that continues to evolve but still moves forward, addressing the complex nature of qualitative research and bringing new insights as we collectively draw a broader picture of EQR.

A Positivist Category

Quality in qualitative research is multidimensional. If quality in quantitative research requires accuracy, precision, rightness, or directness, then quality in qualitative research requires context, locality, properness, and indirectness in addition to those required in quantitative research. This is mainly because qualitative research is value-laden or at least value-related. To help readers better understand our first category, we start with four goals or criteria that are important to consider in the traditional view of EQR. Simply, advocates of this category see qualitative and quantitative research as the same and so use the same criteria, ones based in quantitative research. In a similar vein, mixed-methods scholars identify a series of evaluation criteria necessary for measuring the product and process of mixed methods research ( Dellinger & Leech, 2007 ; Leech, Dellinger, Tanaka, & Brannagan, 2010 ; Tashakkori & Teddlie, 2003 , 2008 ). Sale and Brazil (2004) present a review of criteria for critically appraising mixed-methods research. In their review, they give a very comprehensive list of literature that identifies criteria for evaluating quantitative and qualitative methods in terms of the four conventional validity goals: internal validity, external validity, reliability, and objectivity .

Lincoln and Guba’s (1985) Alternative Category

Perhaps the field of EQR would not be as advanced without Lincoln and Guba’s (1985) alternative approach to judging qualitative research. This approach is well known and, as noted earlier, is still greatly influencing the discourse on EQR. In addressing the traditional goals or criteria of internal validity, external validity, reliability, and objectivity seen in the first category, Lincoln and Guba propose credibility, transferability, dependability, and confirmability , respectively. In Table 32.1 , we briefly explain these parallel goals ( Thomas & Magilvy, 2011 , pp. 152–154).

A “Subtle-Realist” Category

The subtle-realist approach is pragmatic in nature. British scholars Hammersley and Atkinson (1995) and Seale (1999) make a strong case for the necessity of compromise between various extremes. Their philosophical stance in this regard lies between idealism and realism, claiming that neither of them properly addresses the continuing tension of contemporary research, particularly in ethnography. Seale notes, “The widespread appeal of alternative conceptions of research is based upon some fundamental dissatisfactions with the scientific world view” (p. 7). Those who reside in this camp of thought believe that quality in qualitative research is “a somewhat elusive phenomenon that cannot be pre-specified by methodological rules” (p. 7). That is, those concerned with quality in qualitative research don’t necessarily “give up on scientific aims as conventionally conceived, but also draw on the insights of postscientific conceptions of social research” (Seale, p. x). For them, objectivism is seen as “a resource that can be used productively as an attitude of mind by social researchers” (p. 25). Consequently, the discourse on EQR is not fixed but “open to the possibility that conclusions may need to be revised in the light of new evidence” (p. x). A subtle-realist category that is conceptualized in this pragmatic stance is convergent with the following point of view:

Criteriology is, at root, an impossible project if it is intended to reflect an internally logical line of argument that simultaneously reconciles philosophical and political positions with the great variety of research practices which people may wish to pursue. The challenge appears to be to construct some general account of what we might hope to find in a good study that is, on the one hand, open enough to include this variety, and, on the other hand, not so loosely specified as to be no value in providing guidance. ( Seale, 1999 , p. 47)

The relationship between claim and evidence is a starting point for the subtle-realist approach to EQR. Triangulating data, in itself, cannot warrant the credibility of a research report; although triangulation is useful to consider, subtle realists argue that “member validation offers a method for testing researcher’s claims by gathering new evidence” ( Seale, 1999 , p. 71). The quality of qualitative research results from the degree of members’ involvement, whether weak or strong. Thus, openness to the possibility that conclusions may need to be revised in the light of new evidence is determined by the extent to which members are involved in the closeness between evidences and claims.

A General EQR Category

As Seale (1999) noted, a dilemma exists for EQR: the field needs a set of criteria broad enough to include a variety of qualitative research traditions. The field of qualitative research is broad in history, paradigms, theories, and practices. Each qualitative research tradition has its own rationale for quality considerations ( Creswell, 2006 ). Although discipline-specific criteria for these research traditions are available, a majority of the literature on EQR attempts to provide general criteria or validity applicable to qualitative research generally. These attempts are likely to be encountered in many research articles, some of which will be discussed in the next section of this chapter. We define this attempt as belonging to a general EQR category that proposes evaluative guidelines intended to assist reviewers or committee members in judging the quality of qualitative research of any type. It could be seen as too general for some particular types of qualitative research and perhaps too specific for others.

A Category of Post-Criteriology

The post-criteriology category is seen as radical to some extent because those who reside in this category believe that it is neither desirable to use validity or criteria from the conventional positivist standpoint nor even possible to set up predetermined criteria for qualitative research that uncovers complex meaning-making processes.

Is it possible to devise a set of goodness criteria that might apply to an inquiry regardless of the paradigm within which it was conducted? Or is it the case... that goodness criteria are themselves generated from and legitimated by the self-same assumptions that undergrid each inquiry paradigm, and hence are unique to each paradigm? (Guba, 1988, p. 16, cited in Smith, 1990 , p. 168)

Smith (1990) reviewed three alternative paradigms and criteria—post-empiricism or post-positivism, constructivism, and critical theory—and found an overall regulative ideal for inquiry: “objectivity, solidarity, and emancipation,” (p. 183) respectively. His criticism is focused on the assumption that “each paradigm has dispensed with the idea of an absolutely authoritative foundation for knowledge. This nonfoundationalism greatly complicates the criteria issue” (p. 183). There are at least three points common to these different perspectives. First, there is no possibility that a mechanical decision-making procedure can be applied to distinguish valid from invalid research. Second, methodology or procedures, in and of themselves, are not sufficient for making decisions about the quality of inquiry. Finally, although only briefly noted earlier, an appeal to consistently successful prediction is not a live option, in that none of the three perspectives has done very well in this area.

A Post-Validity Category

Before explaining this last category, clarifying the difference between a general sense of credibility used in qualitative research and the theoretical sense of validity used in this section is needed. All the earlier five categories of EQR are more or less direct, straightforward, or less abstract in suggesting ways of judging quality or goodness criteria on qualitative research. The post-validity category has its roots in Patti Lather’s (1986) seminal article, “Issues of Validity in Openly Ideological Research: Between a Rock and a Soft Place,” in which she redefines goodness criteria in ways that make evaluation meaningful for value-based research programs such as feminist research, neo-Marxist ethnography, and Freirian empowering research. She argues that for these research programs to be properly assessed, goodness criteria such as triangulation, construct validity, face validity, and catalytic validity must be built into research designs. That is, critical research programs need accurate data credibility, a researcher’s systematized reflexivity, respect for participants’ interpretation on data (called member-checking), and evidence of participants’ consciousness change.

Later, Scheurich’s (1996) article, entitled “The Mask of Validity: A Deconstructive Investigation,” takes Lather’s value-based research programs a step further, arguing that the conventional approach and Lincoln and Guba’s naturalistic approach are fundamentally similar. That is, the general techniques Lincoln and Guba invented have the same orthodox voices that originated in the positivist paradigm. Social transformational research is validated in ways that require a celebration of the play of multiplicity and difference in data collection, analysis, and interpretation. All in all, EQR in this regard is subject to locality or contextuality, in which meaning is de- or reconstructed toward social justice.

Different Strategies for EQR

Here, having reviewed our sketch of the six general categorical approaches to EQR, we present a series of common strategies for qualitative research evaluation. From the many possible, we select five major strategies for EQR that are different in form and content from one another. In the first, scholars develop a list of criteria or checklist that follows a series of research procedures. In the second, a professional organization sets a high level of research standards. In the third, a reviewer is provided with a rubric or scoring guide to review a journal article. In the fourth, an analysis tool is used to evaluate key aspects of the process and the product of qualitative research. And in the last, we include a set of criteria against which art-based research and performance studies are evaluated.

Ten Commandments

How does one evaluate dissertation studies or journal articles? We find the following list a very typical set of criteria ( Cobb & Hagemaster, 1987 ). We’ll call these the ten evaluative commandments :

Problem and/or research question

Literature review

Data collection

Data processing and plans for analysis

Human subject

Importance to the field

To our knowledge, almost all researchers, scholars, and teachers took an introductory class to learn how to conduct research ( Ambert, Adler, Adler, & Detzner, 1995 ; Burns, 1989 ; Duncan & Harrop, 2006 ; Elliott, Fischer, & Rennie, 1999 ; Forchuk & Roberts, 1993 ; Greenhalgh, 1997 ). What students usually learn is that research goes through a process something like problems → questions/purposes → literature review → context/setting → sample/participants → data collection/display/analysis/interpretation → significance of research. Additionally, students learn about the human subject review process. Reviewing a research project in light of typical research procedures and components is common ( Popay, Rogers, & Williams, 1998 ; Yin, 1999 ). The following review guideline is used in The Asian Journal of Educational Research and Synergy, and it highlights a typical research process using key evaluative criteria (this journal accepts both quantitative and qualitative research):

General Considerations 1. Importance and interest to the journal’s readers • What does the paper contribute to the field of education? • Is it significant to the target community? • Does it present a new and significant contribution to the literature? • Is it timely and relevant? 2. Originality of the paper • Is the study innovative? Interesting? 3. What were the author(s) trying to accomplish and were they successful? Specific Considerations 1. Presentation • Does the paper present a cohesive argument? • What is the basic logic of the presentation? • Are the ideas clearly presented? 2. Writing • Is the writing concise and easy to follow? 3. Length • What portions of the paper should be expanded? Removed? Condensed? Summarized? Combined? 4. Title • Is the title informative? 5. Abstract and introduction • Do the abstract and introduction accurately reflect the points made in the paper? 6. Literature review • Are the cited articles/papers current? • Is the literature review comprehensive? • Does the literature review contain a coherent argument supported by literature (as opposed to a list of studies)? 7. Methods for studies involving primary data collection • Does the author provide enough detail of the methodology? • Are the methods described clearly enough to facilitate replication (where applicable)? • Is there a sound research methodology? • Are the methods appropriate? 8. Data presentation • Could the design be conveyed more easily? • Are the data clearly presented? • Can the reported results be verified easily by reference to tables and/or figures? • Would another form of presentation help? • Are illustrations instructive? • Are all tables and figures clearly labeled? Necessary? Well-planned? 9. Analysis and interpretation • Does the organization of results promote understanding? • Are the analyses appropriate and logical? Are they described in enough detail? 10. Discussion • Are the discussion and conclusions made by the author supported by the data? • Does the writer understand the limitations of his or her work? • Is there enough breadth and depth in the implications of his or her study?

This detailed guideline is intended to help a reviewer examine a journal article and is similar to the ten evaluative commandments presented earlier. We find two considerations interesting in this guideline: originality and discussion. The discussion part covers conclusion, limitations, and implications, all of which are worth being assessed. The originality part, expressed as innovative or interesting , is definitely something important for the reviewer to consider. Arguably, those concerned with a general set of criteria are interested in constructing a checklist inherent in logic, specificity, or thoroughness in form and content. In other words, this kind of checklist-type evaluation strategy is appreciated on the grounds that any research can be assessed in a way that follows a linear sense of logic, specificity, and thoroughness. The next is an example of a review checklist by Clive Seale (1999) , who wrote a seminal book about evaluating the quality of qualitative research. Seale organizes his major checklist items in terms of introduction (two criteria), methods (five criteria), analysis (six criteria), presentation (six criteria), and ethics (one criterion), along with an additional thirty-six subcriteria following these major criteria:

Criteria for the evaluation of qualitative research papers 1. Are the methods of the research appropriate to the nature of the question being asked? 2. Is the connection to an existing body of knowledge or theory clear? Methods 3. Are there clear accounts of the criteria used for the selection of subjects for study and of the data collection and analysis? 4. Is the selection of cases or participants theoretically justified? 5. Does the sensitivity of the methods match the needs of the research questions? 6. Has the relationship between fieldworkers and subjects been considered, and is there evidence that the research was presented and explained to its subjects? 7. Was the data collection and record keeping systematic? Analysis 8. Is reference made to accepted procedures for analysis? 9. How systematic is the analysis? 10. Is there adequate discussion of how themes, concepts, and categories were derived from the data? 11. Is there adequate discussion of the evidence both for and against the researcher’s arguments? 12. Have measures been taken to test the validity of the findings? 13. Have any steps been taken to see whether the analysis would be comprehensible to the participants, if this is possible and relevant? Presentation 14. Is the research clearly contextualized? 15. Are the data presented systematically? 16. Is a clear distinction made between the data and their interpretation? 17. Is sufficient of the original evidence presented to satisfy the reader of the relationship between the evidence and the conclusions? 18. Is the author’s own position clearly stated? 19. Are the results credible and appropriate? Ethics 20. Have ethical issues been adequately considered?

To elaborate, under the Methods heading, Seale (1999) addresses typical issues related to procedures, such as the selection of subjects, theoretical sampling, the relationship between fieldworkers and subjects, and systematic ways of data collection and record keeping. Under the heading of Analysis, he points out basic steps to follow: data analysis procedures (reliability); a degree of systematic analysis; adequate discussion of themes, concepts, and categories; negative case analysis; validity; and checking meaning with respondents. Last, the heading of Presentation discusses a synthesis of data that indicates context-specific, systematic data display; proper interpretation; evidence-based conclusion; the researcher’s position; and credible results. Some subcriteria are: Could a quantitative approach have addressed the issue better? To what extent are any definitions or agenda taken for granted, rather than being critically examined or left open? Has reliability been considered, ideally by independent repetition? Has the meaning of their accounts been explored with respondents? Are quotations, fieldnotes, etc. identified in a way which enables the reader to judge the range of evidence used? Have the consequences of the research... been considered?

Research Standards and Descriptive/Prescriptive Rating Scales

A rigorous attempt to identify a set of general checklist criteria embedded in a linear sense of logic, specificity, and thoroughness is clearly evident in the recent publication of the American Educational Research Association’s (AERA) (2006) Standards for Reporting on Empirical Social Science Research. AERA uses the word standards and organizes its checklist under two overarching themes, warrantability and transparency. Table 32.2 is an excerpt of the AERA research standards, showing the great emphasis placed on analysis and interpretation.

The general research standards in the left column deal with reliability, analysis methods, inference, and conclusion. The specific standards for qualitative research in the right column are focused largely on analysis and interpretation; they are strongly geared toward “being transparent” in the process of developing the descriptions, claims, interpretations, evidence that serves as a warrant for each claim, practices used to develop and enhance the warrant for the claims, and interpretive commentary. Presumably, these two core themes, warrantability and transparency, proclaimed by the world’s largest educational research association, have significant impact on the qualitative research community in many ways. Warranted claims and transparent procedures could be construed as political in nature and have been used in recent years in the name of scientific, evidence-based research by political conservatives typically thought to oppose the use and funding of qualitative research ( Denzin, 2012 ). However, many qualitative researchers appear to endorse the word transparency as a newly emerging and important criterion in conducting and evaluating qualitative research.

Table 32.3 is a review form for evaluating AERA annual conference proposals. It addresses the research standards alluded to earlier by specifying warrantability and transparency. The evaluation contents or criteria of this review form are aligned with general research procedures, just like those of checklists, but they are much more descriptive and prescriptive. The form describes what each research component is like (e.g., perspectives or theoretical framework) and, at the same time, it prescribes what must be expected by a reviewer (e.g., evidence, substantiation or warrants for arguments, and scientific significance). Additionally, it gives a 1–5 rating scale. Typically, reviewers are eventually asked to make a decision. To our knowledge, providing written comments is typical, along with stating a decision that falls within one of four judgmental calls: accepted as is, accepted with minor revision, accepted with major revision, or rejected. The AERA proposal review evaluation form has a binary decision rule—accepted or rejected—and includes comments for both writer and division chair.

Evaluative Rubrics

Table 32.4 is a rubric-type review form for the journal Multicultural Perspectives . This journal accepts both quantitative and qualitative work, but mostly includes qualitative research articles.

This evaluation rubric reviews journal articles in the context of multiculturalism (race, gender, ethnicity, etc.); because multicultural education includes several dimensions that deal with general thematic criteria , these differ from generally encountered criteria like questions, purposes, literature, analysis, and conclusion. Given a number of different notions of multiculturalism and multicultural education, this journal’s evaluation rubric adopts such general thematic criteria as provocative content (new and thought-provoking) and organized/focused, clear/comprehensive, or interesting reading, along with commonly addressed criteria such as significant topic, clear purpose/scope and methods, and appropriateness to the journal. This review rubric, or general thematic rubric, with its nine dimensions/criteria, not only assists reviewers in evaluating broad ranges of research articles submitted to this interdisciplinary journal, but also seeks a high level of article quality by emphasizing strong qualitative evaluation criteria (e.g., “new and thought-provoking”).

Directions: Place an “X” for each dimension: Ex = Excellent; G = Good; M = Marginal; W = Weak. Jot notes in the “comments” section and incorporate these into the narrative.

Criteria: By, For, and Of the Readers, Participants, and Investigators

In the matter of evaluating content and form, we have thus far examined a series of criteria set forth in checklists, standards, and rubrics. We would like to draw attention to another, different form of evaluation. If the previous strategies and discussions on determining the inclusion of evaluation criteria are straightforward and directive in terms of what qualitative research is like and how it proceeds, then the argument that Stiles (1999) makes is insightful and relational:

The concept of objectivity is replaced by the concept of permeability , the capacity of understanding to be changed by encounters with observations. Investigators argue that we cannot view reality from outside of our own frame of reference. Investigator bias can be reframed as impermeability. ... Good practice in reporting seeks to show readers how understanding has been changed. The traditional goal of truth of statement is replaced by the goal of understanding by people . Thus, the validity of an interpretation is always in relation to some people, and criteria for assessing validity depend on who that person is (e.g., reader, investigator, research participant). (p. 99; emphasis in original)

To elaborate, according to Stiles (1999) , EQR involves two sets of judgments on quality: good practice criteria and validity criteria . Here, we briefly explain the first: judgmental quality criteria. It is likely that all sorts of criteria mentioned in the previous types of evaluation thus far are convergent with what Stiles refers to as good practice criteria in light of the investigator’s choice, sound analytical practices, and disclosures of the investigator’s forestructure. Some example criteria include: “Are research questions clearly stated? Are prolonged and persistent observation made? Did the investigator make a disclosure of his or her orientation or assumptions?” (p. 99). These judgmental criteria and their subcriteria are intended to evaluate the degree of what is generally called “credibility” or claims of truthfulness.

What makes Stiles’s (1999) strategy unique in the matter of EQR is the “validity criteria” (p. 100) that are mainly concerned with who is impacted by the researchers’ interpretations and how the impact of interpretation is utilized and for what purpose. The table of analytic evaluation developed by Stiles is seen in Table 32.5 .

The 3×2 grid analysis tool in Table 32.5 involves three different stakeholders and two different purposes of interpretation. For example, if the purpose of interpretation is to determine readers’ agreement with regard to what is found in the research, then one major criterion should be coherence , which includes follow-up questions like “Is the interpretation internally consistent? Is it comprehensive?... Does it encompass all of the relevant elements and the relations between elements?” (p. 100). If the purpose of interpretation is to make readers rethink their existing belief system, then they should have revealing or self-evident learning experiences as they read a text. Subquestions related to this level of evaluation include “Is the interpretation a solution to the concern that motivated the reader’s interest?... Did it produce change or growth in the reader’s perspective? Did it lead to action?” (p. 100).

At the level of evaluation criteria to be applied to research participants, the major criterion is testimony , which allows participants to express their voices from their own perspectives. Follow-up questions are “Did participants indicate that the interpretation accurately described their experience?... Were their reactions to hearing the interpretation consistent with the interpretation’s motifs? Did they reveal fresh and deeper material?” ( Stiles, 1999 , p. 100). Catalytic validity , one of Guba and Lincoln’s (1989) five authenticity criteria, is used if the purpose of interpretation is to empower the participants’ life worlds and to have them “take more control of their lives” (p. 100). This catalytic validity is also more purposefully and critically used in emancipatory social science research (e.g., feminist research, neo-Marxist critical ethnography, and Freirian research). In effect, Lather (1986) radically redefines catalytic validity as indicating “not only... a recognition of the reality-altering impact of the research process itself, but also... the need to consciously channel this impact so that respondents gain self-understanding and, ideally, self-determination through research participation” (p. 67).

Criteria for Art-Based Research and Performance Studies

In recent years, art-based researchers have proposed six evaluative criteria. Barone and Eisner (2012) note with some critical comment on existing inquiry into EQR that “employing a quantitative metric enables one to enumerate or to summarize quantity.... Criteria [for arts-based qualitative work] are much more slippery” ( Barone & Eisner, 2012 , p. 147). With specific art-based evaluative criteria in mind, they propose the following set of criteria:

Incisiveness : The degree to which research gets to the core essence of a social issue; Barone and Eisner (2012) assert that incisive research: “offer[s] the potential for waking the reader up to a strange world that appears new and yet always existed in the shadowy corners of the city that they had never explored on their own” (p. 149)

Concision : The degree to which research occupies the minimal amount of space; “any additional material simply diminishes the capacity of the piece to achieve that purpose, waters down the power of the work, and hence its effectiveness” (pp. 149–150)

Coherence: The creation of a work of arts-based research whose features hang together as a strong form (pp. 150–151)

Generativity: The ways in which the work enables one to see or act on phenomena, even though it represents a kind of case study with an n of only 1 (pp. 151–152)

Social significance : Something that matters, ideas that count, important questions to be raised (p. 153)

Evocation and illumination : Feeling or defamiliarizing an object so that it can be seen in a way that is entirely different from the ways in which customary modes of perception operate (p. 154)

Barone and Eisner (2012) add that these six criteria should be seen as “a cue for perception” (p. 154), one that assists observers or audiences in making a better evaluation of an art product. Therefore, they offer these criteria merely as a starting point for thinking about the appraisal of works of art-based research. Getting locked into criteria that constrain innovation and dampen imagination is undesirable. As with the other scholars mentioned earlier, Barone and Eisner take a deliberative, balanced perspective on EQR. Barone and Eisner assert,

We do not believe that we can have an effective arts based research program without some degree of common reflection over what might be attended to in looking at such work. Thus, in a certain sense, we compromise between, on the one hand, common criteria and, on the other, criteria that are idiosyncratic to the work itself. This may appear a dilemma, but it is a reality. (p. 155)

The compromise alluded to is indeed a reality, one that those involved in qualitative research deal with regularly. Seeing qualitative research as art is not new. But following a “recipe” to produce art-based research is like using a recipe to produce a chocolate cake to a particular standard. The problem is that “the more detailed and prescriptive the recipe, the more likely that the cakes made from that recipe will be indistinguishable from one another” ( Barone & Eisner, 2012 , p. 155). Eventually, Barone and Eisner “invite you, the readers, to use your own judgment in applying these criteria to the examples of the works of arts based research” (p. 155).

Cho and Trent (2009) suggest validity criteria for assessing performance-related studies. Performance is often viewed as an “object” or the presentation of the results of analysis ( Hamera & Conquergood, 2006 , p. 420). In this view, qualitative researchers think, plan, select, and show through performance their inquiry findings as the last phase of assignment/research project completion. Although Cho and Trent support this traditional role for performance in qualitative research, they claim that their conceptualization is broader, incorporating performance aspects at all stages of the inquiry process. They acknowledge the meaning of performance both in and as qualitative research because the boundary between performance and qualitative research blurs as researchers/teachers and students/audience or researchers and reviewers come to see “conducting qualitative research” as an inevitably personal, social, and political performative process. They advocate for in-depth dialogues and scaffolding to support audiences’ and other researchers’ introduction to the possibility of constructing and utilizing performance in/as qualitative inquiry ( Hamera, 2006 ).

Cho and Trent (2009) offer validity criteria for performance that are critically oriented, culturally responsive, and pedagogically sound. The rubric they construct is not only evaluative but also pedagogical in nature (see Table 32.6 ). The rubric outlines criteria for all three phases of the performative process: pre-, during, and post-performance.

Pre-performance as imaginative rehearsal is an ongoing textual rehearsal process as the researcher finalizes the analysis and interpretation of the data collected. The focus of imaginative rehearsal is on making the voices of subjects relational and evocative as the researcher constructs texts as scripts. Criteria needed to evaluate this imaginative textual practice involve data sufficiency, level of critical interpretation, and degree of script craftsmanship. The stage of performance-in-use, associated with artistic re/presentation, involves transacting the lived experiences of others with audiences by means of the voices and bodies of the performer(s). One of the main criteria is degree of understandability of the performance being re/presented. With clear delivery in mind, this criterion is one that cautions that some performance is too complex to understand. The post-performance stage is nurtured by a co-reflexive member-checking process among subjects, performers, and audience. It is important to link artistic re/presentation with degrees of intensive experience and closeness between the performer and the audience. Post-performance is seen as a beginning, not an ending, because the effect of a performance on the performer and the audience may be rearranged as both parties share their understandings with one another. The performer should be very clear about his or her rationale for checking validity: Whose authority? Whose artistic achievements? And, whose evaluative validity is of most importance at this time in this place? Which choices promote the primary aim of attaining a deeper, empathic understanding across participants (both performers and audience members)? These co-constructive validity-seeking questions may help audiences reflect critically, not so much on aesthetics at the surface level as on hidden messages underpinning the performance.

Evaluating Qualitative Research: Politics of Evidence for the Twenty-First Century

The criteria for judging a good account have never been settled and are changing. ( Clifford, 1986 , p. 9)

The question of whether it is possible to measure the value of qualitative research from the standpoint of conventional evaluation criteria has resurfaced. Those who accept a positivist paradigm assume that reality can be objectively measured. These researchers are reigniting the paradigm wars in ways that repeat old arguments in new form. The new focus of the attack is on the shaky nature of evidence drawn from qualitative research. As has been explored throughout this chapter, much scholarship has been focused on EQR in recent decades. As a consequence, more accurate, meaningful ways of evaluating qualitative research have been established.

Despite the evolution of robust evaluation frameworks, work remains on at least two fronts. Internally, as a community of qualitative researchers, we need to continue to focus on the purposes of our scholarly work and the ways we legitimize it both within and outside our fields. This, necessarily, is a never-ending conversation, and one in which all researchers should participate. Externally, we need to continue to focus on appropriate responses to those who diminish the rigorously obtained knowledge that results from naturalistic inquiry. Those who prioritize only randomized, generalizable work with numerical findings (despite the inherent associated problems) ignore a robust knowledge base that, pedagogically, has often more to offer than a statistical analysis of decontextualized “data.” This knowledge base presents in narrative form, as stories, and, as humans and inquirers, it is among our most basic ways of knowing. Unfortunately, as we discuss later in this chapter, those who perpetuate paradigm wars also wield a great deal of power in research and policy communities.

Evaluating Qualitative Research: Moving Forward in Contemporary Contexts

Thus far, our review has illuminated the wide variety of approaches to EQR, including wide-ranging epistemological underpinnings, as well as a broad array of strategies and processes for evaluating qualitative work. Still, despite extant models and frameworks, researchers work in dynamic, always changing contexts—socially, personally, and politically. As noted earlier, there is and always will be a need to continue to examine emergent evaluation prescriptions and proposals and to juxtapose these with contemporary evolutions in context and culture. Richardson’s evolving work on this topic provides a good example. Richardson, in 2000, offered five criteria against which to assess the validity/quality of ethnographic texts:

Substantive contribution : Does this piece contribute to our understanding of social life? Does the writer demonstrate a deeply grounded (if embedded) human-world understanding and perspective? How has this perspective informed the construction of the text?

Aesthetic merit : Does this piece succeed aesthetically? Does the use of creative analytical practices open up the text, invite interpretive responses? Is the text artistically shaped, satisfying, complex, and not boring?

Reflexivity : How did the author come to write this text? How was the information gathered? Ethical issues? How has the author’s subjectivity, as both a producer and a product of this text, been addressed? Is there adequate self-awareness and self-exposure for the reader to make judgments about the point of view? Do the authors hold themselves accountable to the standards of knowing and telling of the people they have studied?

Impact : Does this affect me? Emotionally? Intellectually? Generate new questions? Move me to write? Move me to try to new research practices? Move me to action?

Express a reality : Does this text embody a fleshed out, embodied sense of lived experience? Does it seem “true” —a credible account of a cultural, social, individual, or communal sense of the “real”? (p. 254)

The evaluative questions listed here concerning ethnographic texts can be applied to judging most qualitative texts. As a journal referee, one must be concerned with the degree of contribution, a sense of aesthetics, the level of a researcher’s reflection, the learning of the reader, and indications of credibility. It appears, however, that Richardson’s criteria have changed. When writing with St. Pierre in 2005, Richardson and St. Pierre exclude the last criterion, express a reality . There is no explanation as to why the last criterion about credibility is no longer included in this later version.

It is typical for research methodologists to offer a set of evaluative criteria that are claimed to be relevant and necessary based on their theoretical underpinnings. Many of the scholars highlighted in this chapter have done so, and these criteria sets illustrate that some criteria are commonly used, whereas other criteria are used uniquely, depending on the different purposes and uses of the evaluation. Yet, by looking at the matter of EQR from a broader perspective, we may end up concluding that EQR, like other theoretical constructions in social science, is simultaneously contextual, cultural, and political. When a reviewer evaluates a manuscript, the process is individualistic, and it is hard to describe the multiple influences impacting the reviewer’s perspective. These individualistic and hidden meanings used by a reviewer do not necessarily match neatly with a set of criteria provided by a journal editor, colleague, or conference organizer. Assessment tools in this complex process are used for formality, convenience, and as a standardized means to ensuring fairness in determining contributors. In the end, it is the reviewer’s construction of meaning (or lack of) around the text that matters.

By the same token, an inclusion or exclusion of goodness criterion is socially constructed. The earlier noted discrepancy between Richardson (2000) and Richardson and St. Pierre (2005) serves as an example. The omission must be more than random. The co-authors likely included those criteria on which they agreed and co-constructed understandings, and omitted those on which they did not. A consistent theme across both authors’ individual and collaborative work is the joining of art and science in the production of qualitative texts. “Science is one lens, and creative art is another. We see more deeply using two lenses. I want to look through both lenses to see ‘a social science art form’ —a radically interpretive form of representation” (Richardson & St. Pierre, p. 964). Perhaps the qualitative research community accepts these scholars’ social science art form, which is similar to what Lather (1986) refers to as “a new rigor of softness... validity of knowledge in process... an objective subjectivity” (p. 78). A social science art form or an objective subjectivity is something that continues to evolve. A constant deliberation on the inclusion and exclusion of criteria in EQR is necessary to better address the changing nature of knowledge and aesthetics in sociocultural contexts.

Evaluation, Criteria, and Power

Scholars continue the conversation about evaluating research. Tracy (2010) presents a recent proposal for a model to ensure “excellent qualitative research.” Tracy’s model is a solid synthesis of what has been researched and theorized about in recent history. Alternatively, Lichtman’s (2006) review of evaluating qualitative research includes personal criteria , which are based on her philosophy and assumptions regarding a good piece of qualitative research. Thus, Lichtman attempts to make her personal philosophy explicit by reflecting on the self, the other, and interactions of the self and other. Lichtman argues that “an understanding of the other does not come about without an understanding of the self and how the self and other connect” (p. 192). Then she goes on, “I believe each is transformed through this research process” (p. 192). In contrast, Tracy takes an objective stance in establishing her model’s rationale for education establishment power holders:

In addition to providing a parsimonious pedagogical tool, I hope my conceptualization may aid in garnering respect for qualitative methods from power holders who know little about our work. Despite the gains of qualitative research in the late 20th century, a methodological conservatism has crept upon social science over the last 10 years... evidenced in governmental and funding agencies’ preference for research that is quantitative, experimental, and statistically generalizable.... High ranking decision makers—in powerful governmental, funding, and institutional review board positions—are often unprepared and unable to appropriately evaluate qualitative analyses that feature ethnography, case study, and naturalistic data. ( Tracy, 2010 , pp. 837–838)

With these pedagogical and political purposes in mind, Tracy (2010) provides eight universal hallmarks for high-quality qualitative methods across paradigms, suggesting that each criterion of quality can be approached via a variety of paths and crafts, the combination of which depends on the specific researcher, context, theoretical affiliation, and project. Her eight “big-tent” criteria for excellent qualitative research are listed in Table 32.7 .

We’ll examine two of these criterion for clarification: “rich rigor” and “meaningful coherence.” The nature of rigor is tricky and difficult for evaluators to define. Rigor in qualitative research differs from that in quantitative research. Rigor literally means “stiffness,” from the Latin word rigere, to be stiff, and it implies rigidity, harshness, strict precision, an unyielding quality, or inflexibility. The term qualitative rigor, then, is an oxymoron, considering that qualitative research is “a journey of explanation and discovery that does not lead to stiff boundaries” ( Thomas & Magilvy, 2011 , p. 151). Thus, the word rigor involves many dimensions that must be considered. In qualitative research, rigor often refers to the thorough, ethical conduct of a study of a social phenomenon. We argue that all criteria—rigor and numerous others—used (or considered) in evaluating qualitative research are necessary but may not be sufficient. Tracey’s (2010) thesis, therefore, is in line with the tricky nature of rigor, which also reflects what Richardson mentioned earlier, a wish to have a social science art form of EQR:

Like all components in this conceptualization—rich rigor is a necessary but not sufficient marker of qualitative quality. For qualitative research to be of high quality, it must be rigorous. However, a head full of theories and a case full of data does not automatically result in high quality work. Qualitative methodology is as much art as it is effort, piles of data, and time in the field. And just like following a recipe does not guarantee perfect presentation, or completing a vigorous training plan does not guarantee race-day success, rigor does not guarantee a brilliant final product. That being said, rigor does increase the odds for high quality, and the methodological craft skills developed through rigorous practice transcend any single research project, providing a base of qualitative fitness that may enrich future projects. Table 32.7 Eight “big-tent” criteria for excellent qualitative research Criteria for quality .  Various means, practices, and methods through which to achieve (end goal) .  Worthy topic The topic of the research   • Relevant • Timely • Significant • Interesting Rich rigor The study uses sufficient, abundant, appropriate, and complex   • Theoretical constructs • Data and time in the field • Sample(s) • Context(s) • Data collection and analysis processes Sincerity The study is characterized by   • Self-reflexivity about subjective values, biases, and inclinations of the researcher(s) • Transparency about the methods and challenges Credibility The research is marked by   • Thick description, concrete detail, explication of tacit (nontextual) knowledge, and showing rather than telling • Triangulation or crystallization • Multivocality • Member reflections Resonance The research influences, affects, or moves particular readers or a variety of audiences through   • Aesthetic, evocative representation • Naturalistic generalizations • Transferable findings Significant contribution The research provides a significant contribution   • Conceptually/theoretically • Practically • Morally • Methodologically • Heuristically Ethical The research considers • Procedural ethics (such as human subjects) • Situational and culturally specific ethics • Relational ethics • Exiting ethics (leaving the scene and sharing the research) Meaningful coherence The study • Achieves what it purports to be about • Uses methods and procedures that fit its stated goals • Meaningfully interconnects literature, research questions/foci, findings, and interpretations with each Criteria for quality .  Various means, practices, and methods through which to achieve (end goal) .  Worthy topic The topic of the research   • Relevant • Timely • Significant • Interesting Rich rigor The study uses sufficient, abundant, appropriate, and complex   • Theoretical constructs • Data and time in the field • Sample(s) • Context(s) • Data collection and analysis processes Sincerity The study is characterized by   • Self-reflexivity about subjective values, biases, and inclinations of the researcher(s) • Transparency about the methods and challenges Credibility The research is marked by   • Thick description, concrete detail, explication of tacit (nontextual) knowledge, and showing rather than telling • Triangulation or crystallization • Multivocality • Member reflections Resonance The research influences, affects, or moves particular readers or a variety of audiences through   • Aesthetic, evocative representation • Naturalistic generalizations • Transferable findings Significant contribution The research provides a significant contribution   • Conceptually/theoretically • Practically • Morally • Methodologically • Heuristically Ethical The research considers • Procedural ethics (such as human subjects) • Situational and culturally specific ethics • Relational ethics • Exiting ethics (leaving the scene and sharing the research) Meaningful coherence The study • Achieves what it purports to be about • Uses methods and procedures that fit its stated goals • Meaningfully interconnects literature, research questions/foci, findings, and interpretations with each Open in new tab ( Tracy, 2010 , p. 841; emphasis in original)

Tracy (2010) uses metaphors of art and recipes to point out that a claim for rigor involves a closer investigation . Its promise and limitations coexist. The politics of “being rigorous” is clearly evident in many types of qualitative research. Likewise, techniques to ensure “rigor,” such as advanced statistical analyses, do not guarantee brilliant quantitative research, either. It is the perception of reviewers or assessors that decides what makes research “good research.” All judgment calls involves a complex mix of relative, contextual, political, and/or ethical criteria. In this regard, “tools, frameworks, and criteria are not value free” ( Tracy, 2010 , p. 838).

Meaningful coherence, Tracy’s final criterion, is accomplished when “the study achieves what it purports to be about, uses methods and procedures that fit its stated goals, and meaningfully interconnects literature, research questions/foci, findings, and interpretations with each” (p. 839). Thus, this criterion is likely to be seen as a summary of overall judgments in a typical evaluation tool.

Tracey’s “big-tent” set of criteria is a synthesis of other scholars’ constructions of existing goodness criteria. These criteria may usefully remind reviewers about a variety of judgmental aspects in their attempts to determine “how good is good enough,” but it is also important to think about the fact that qualitative research “should not be mechanically scored and summed insofar as some issues may be far more important than others in particular studies” ( Stiles, 1999 , p. 100). In the end, it is necessary to develop some kind of standardized form of evaluative criteria to be used in qualitative research. Such constructions provide us with meaningful evaluation tools or guidelines, aligned with key criteria, which determines the degree of credibility in qualitative research. Yet, is it really possible to develop standardized forms of evaluation applicable to any type of qualitative research? Tracy (2010) thinks it is:

Perhaps the most controversial part of this conceptualization is the notion of universal criteria for qualitative quality. However, I believe that we need not be so tied to epistemology or ontology (or the philosophy of the world) that we cannot agree on several common end goals of good qualitative research. Qualitative methodologists range across postpositivist, critical, interpretive, and poststructural communities. In contrast,... researcher reflexivity is a validity procedure clearly positioned within the critical paradigm where individuals reflect on the social, cultural, and historical forces that shape their interpretation..., I would argue instead that researcher reflexivity—like many other practices for goodness—serves as an important means toward sincerity for research in a number of paradigms. Its utility need not be bound only to critical research. ( Tracy, 2010 , p. 849)

Nonetheless, we find that some prestigious qualitative journals don’t provide these kinds of criteria or guidelines for their reviewers. Instead, reviewers invited by these journals are provided with very general guidelines. Table 32.8 is an example of the International Journal of Qualitative Studies in Education (QES) review form.

As Table 32.8 shows, there are no specific criteria to be used in reviewing manuscripts in this prestigious qualitative research journal. Nonetheless, editorial manager Gonzalez (2012) has confidence in this open process: “Reviewers are free to send any comment to the author. We have very strong scholars to agree to review and most of the times, our reviewers are very detailed (without asking them) in their reviews from grammar, to format, to content... many of them go and make comments to each section of the manuscript (intro, methodology, results, conclusions)” (personal communication, August 16, 2012). In this review process, what we find is a sense of autonomy, fit, trust, and professional ethics. Reviewers who have expertise know what is worth assessing and how good is good enough.

Yet, there are external forces that question not only quality in qualitative research but also its legitimacy. For example, mixed-methods scholars and researchers try not to see themselves as post-positivists in the research paradigms that have been well established over the past several decades ( Guba & Lincoln, 1994 ; Lincoln & Guba, 2000 , 2005 ; Lincoln, Lynham, & Guba, 2011 ) but instead seek to create their own hybrid epistemology, one that they prioritize over qualitative research. The current neo-conservative initiatives—the National Research Council (NRC) or the Society for Research on Educational Effectiveness (SREE) (see Denzin, 2009 , for more detail)—diminish the tradition of qualitative inquiry that values understanding in human science by narrowly defining what research is and how it should be assessed. Denzin (2009) points to the necessity of casting big-tent criteria to evaluate qualitative research in the context of a changing epistemological and political context:

[W]e must expand the size of our tent, indeed we need a bigger tent! We cannot afford to fight with one another. Mixed-methods scholars have carefully studied the many different branches of the poststructural tree.... The same cannot be said for the poststructuralists. Nor can we allow the arguments from the SBR [Scientifically Based Research] community to divide us. We must learn from the paradigm conflicts of the 1980s to not over-reach, to not engage in polemics, to not become too self-satisfied. We need to develop and work with our own concepts of science, knowledge and quality inquiry. We need to remind the resurgent postpositivists that their criterion of good work applies only to work within their paradigm, not ours. (pp. 32–33)

As implied here, current discourse on the politics of evidence is mostly a resurrection of old-fashioned epistemological debates, which are initiated from several organizations or councils at the national level in the United States (e.g., NRC, SREE, the Cochrane Clearinghouse, the Campbell Methods Group, or the What Works Clearinghouse). These trends are generally called scientifically based Research (SBR) or evidence-based movement (EBM). The extended discussion goes beyond the scope of this chapter. The main epistemological questions that need to be asked, just as they were forty years ago, are: “Whose science? Whose scientific principles?” ( Denzin, 2009 , p. 141). Related to the inquiry of this chapter, we ask, “Whose criteria?”

Tracy’s (2010) eight “big-tent” criteria and the QSE’s simple scholarly decision recommendation with its open-ended comments are two extreme approaches within our qualitative research community. Those situated in the positivist epistemology and mixed-method scholars will likely prefer Tracy’s (2010) “big-tent” criteria for excellent qualitative research over the QSE’s simple form. This is not because Tracy’s reconstruction of other scholars’ constructions is absolutely truthful or valid in itself, but because Tracy approaches it procedurally, in terms of a logical flow of what a reviewer needs to do. The beauty of “big-tent” procedural criteria is that it is normative, to the extent that a reviewer should not disregard the work of an author due to a disagreement with the author’s epistemology. This also applies to the other evaluative extreme, such as the QSE’s recommendation sheet with open-ended comments, in which a reviewer has the freedom to make a scholarly judgment. In our opinion, the current debate on the politics of evidence is too heavily focused on ideology while giving too little attention to ethical concerns.

In demonstrating methodological excellence, we need to take care of ourselves in the process of taking care of others. The most successful researchers are willingly self-critical, viewing their own actions through the eyes of others while also maintaining resilience and energy through acute sensitivity to their own well-being. ( Tracy, 2010 , p. 849)

Lincoln and Guba’s (1985) constructivist criteria to evaluate our qualitative research processes and products started a rich conversation and decades of scholarship designed to hone and refine those criteria initially proposed and to discover increasingly rich and creative ways to address the challenge of evaluation. Lincoln and Guba argue that trustworthiness is always negotiable, not being a matter of final proof whereby readers are forced to accept an account. Therefore, the field of EQR is not an oxymoron. Much has been known about the nature of evaluative criteria in qualitative research. Some propose a general set of criteria, whereas others focus on specific sets of criteria. This chapter identifies six categories of EQR: (1) a positivist category, (2) Lincoln and Guba’s alternative category, (3) a “subtle-realist” category, (4) a general EQR category, (5) a category of post-criteriology, and (6) a post-validity category. As seen in many strategies or examples of EQR, some make lists of questions about what is commonly expected in assessing the process and product of qualitative research, whereas others select key validity criteria against which an essence of qualitative research is identified, discussed, and evaluated. Still others adopt broad criteria across these different approaches to construct a comprehensive framework.

As evidence in educational research has continued to be more narrowly defined, many qualitative researchers propose clear counterarguments. Efforts will continue in the search for evaluative criteria from inside the qualitative research community, and the field of EQR will continue to grow, theoretically and practically.

What future directions can we expect for EQR in the twenty-first century? As discussed in this chapter, evaluating qualitative research is complex, challenging, and exciting all at the same time. What matters most is accepting this dilemma, celebrating the reality, and creating a holistic storyline (or a common playful intellectual ground) intended to invite those who have diverse backgrounds to bring different evaluative tools toward constructing flexible but firm evaluation theory, policy, and practice. The qualitative research community may do well to pay close attention to Barone and Eisner’s (2012) compromise between common and unique criteria. The beginning of this holistic story has been written, and we hope that others jump in to constructively compete in searching for a common ground in evaluating qualitative research. One of authors of this chapter writes ( Cho, 2010 ):

The shape of a hexagon is naturalistic. Beehives, snowflakes, or molecules are some examples that can be found in nature. We like this hexagon shape just because it seems to represent a balance. A triangle implies a sense of absolute stability or a function of geometric equilibrium. A hexagon shows a sense of balance or harmony particularly when it is connected with others. It looks complicated and messy at a distance but patterned and fabricated when closely seen. Imagine that bees constantly move around the surface of beehive. A beehive is constructed in compactly connected hexagon shapes as bees diligently work with beeswax from their bodies. This analogy can lead qualitative researchers to be more creative in their practical engagement with validity. The shape of a hexagon is unique in that it leads to harmony and balance as it is tightened from, and connected to each other. (p. 4)

EQR is more than a sum of its parts. It goes beyond creating a set of checklists or recipes. Furthermore, it is more than paradigmatic idiosyncrasy. It should be holistic in nature. Our holistic approach to EQR doesn’t seek a complete sense of convergence. Instead, it leaves some room, some unknown territory that may never be reached by the researcher. Like a bee that intuitively and holistically dances around and filters pollen into beeswax to construct a hive, a reviewer deeply imbibes both the process and the product of qualitative research to clearly ensure acceptable quality. Twenty-first century criteria that we support include (1) thought-provoking ideas, (2) innovative methodologies, (3) performative writing, and (4) global ethics and justice-mindedness. Riessman’s (2008) reflection on truths and cautions is our ending in a new beginning:

I prefer not to think in terms of standards or criteria, and warn students away from the “paradigm warfare” that exists out there in the literature. It can paralyze and... simplify what are complex validation and ethical issues all investigators face.... Narrative truths are always partial—committed and incomplete. (pp. 185, 186)

Altheide, D. , & Johnson, J. ( 1994 ). Criteria for assessing interpretive validity in qualitative research. In N. Denzin & Y. Lincoln (Eds.), Handbook of qualitative research (pp. 485–499). Newbury Park, CA: Sage Publications.

Google Scholar

Google Preview

Altheide, D. , & Johnson, J. ( 2011 ). Reflections on interpretive adequacy in qualitative research. In N. Denzin & Y. Lincoln (Eds.), The SAGE Handbook of qualitative research (4th ed.) (pp. 581–594). Thousand Oaks, CA: Sage Publications.

American Educational Research Association. ( 2006 ). Standards for reporting on empirical social science research in AERA publications.   American Educational Research Association Educational Researcher , 35 (6), 33–40.

Ambert, A. , Adler, P. , Adler, P. , & Detzner, D. ( 1995 ). Understanding and evaluating qualitative research.   Journal of Marriage and Family , 57 (4), 879–893.

Barone, T. , & Eisner, E. ( 2012 ). Arts based research . Thousands Oak, CA: Sage Publications.

Burns, N. ( 1989 ). Standards for qualitative research.   Nursing Science Quarterly , 2 (1), 44–52.

Cho, J. (2010). Searching for “good” data: A reflection on validity-in-practice with double-edged problems. Division D - Qualitative Inquiry Invited Symposium: Working validity: Transactional, transformational, and holistic approaches to qualitative inquiry . Paper presented at AERA (American Educational Research Association), May, Denver, CO.

Cho, J. , & Trent, A. ( 2006 ). Validity in qualitative research revisited.   Journal of QualitativeResearch , 6 (3), 319–340.

Cho, J. , & Trent, A. ( 2009 ). Validity criteria for performance-related qualitative work: Toward a reflexive, evaluative, and co-constructive framework for performance in/as qualitative inquiry.   Qualitative Inquiry , 15 (6), 1013–1041.

Clifford, J. ( 1986 ). Introduction: Partial truths. In J. Clifford & G. Marcus (Eds.), Writing culture: The poetics and politics of ethnography (pp. 1–26). CA: University of California Press.

Cobb, A. , & Hagemaster, J. ( 1987 ). Ten criteria for evaluating qualitative research proposals.   Journal of Nursing Education , 26 (4), 138–143.

Corbin, J. , & Strauss, A. ( 2008 ). Basics of qualitative research (3rd ed.). Thousands Oak, CA: Sage Publications.

Creswell, J. ( 2006 ). Qualitative inquiry and research design: Choosing among five approaches (2nd ed.). Thousand Oaks, CA: Sage Publications.

Dellinger, A. , & Leech, N. ( 2007 ). Toward a Unified Validation Framework in Mixed Methods Research.   Journal of Mixed Methods Research , 1 (4), 309–332.

Denzin, N. ( 2009 ). The elephant in the living room: Or extending the conversation about the politics of evidence.   Qualitative Inquiry , 9 (2), 139–160.

Denzin, N. ( 2012 ). Triangulation 2.0.   Journal of Mixed Methods Research , 6 (2), 80–88.

Denzin, N. , & Giardian, M. ( 2009 ). Introduction. In N. Denzin & M. Giardian (Eds.), Qualitative inquiry and social justice (pp. 11–50). Walnut Creek, CA: Left Coast Press, Inc.

Duncan, S. , & Harrop, A. ( 2006 ). A user perspective on research quality.   International Journal of Social Research Methodology , 9 (2), 159–174).

Elliott, R. , Fischer, C. , & Rennie, D. ( 1999 ). Evolving guidelines for publication of qualitative research studies in psychology and related fields.   British Journal of Clinical Psychology , 38 (3), 215–299.

Forchuk, C. , & Roberts, J. ( 1993 ). How to critique qualitative research articles.   Canadian Journal of Nursing Research , 25 (4), 47–56.

Greenhalgh, T. ( 1997 ). Assessing the methodological quality of published papers.   British Medical Journal , 315 (7103), 305–308.

Guba, E. , & Lincoln, Y. ( 1989 ). Fourth generation evaluation . Newbury Park: Sage Publications.

Guba, E. , & Lincoln, Y. ( 1994 ). Competing paradigms in qualitative research. In N. Denzin & Y. Lincoln (Eds.), Handbook of qualitative research (pp. 105–117). Newbury Park, CA: Sage Publications.

Hamera, J. , & Conquergood, D. ( 2006 ). Performance and politics: Themes and arguments. In D. S. Madison & J. Hamera (Eds.), The Sage handbook of performance studies (pp. 419–425). Thousand Oaks, CA: Sage Publications.

Hamera, J. ( 2006 ). Introduction: Opening opening acts. In J. Hamera (Ed.), Opening acts: Performance in/as communication and cultural studies (pp. 1–10). Thousand Oaks, CA: Sage Publications.

Hammersley, M. , & Atkinson, P. ( 1995 ). Ethnography: Principles in practice (2nd ed.). London: Routledge.

Lather, P. ( 1986 ). Issues of validity in openly ideological research: Between a rock and a soft place.   Interchange , 17 (4), 63–84.

Leech, N. , Dellinger, A. , Tanaka, H. , & Brannagan, K. ( 2010 ). Evaluating mixed research studies: A mixed methods approach.   Journal of Mixed Methods Research , 4 (1), 17–31.

Lichtman, M. ( 2006 ). Qualitative research in education: A user’s guide . Thousands, CA: Sage Publications.

Lichtman, M. ( 2009 ). Qualitative research in education: A user’s guide (2nd ed) . Thousands, CA: Sage Publications.

Lincoln, Y. , & Guba, G. ( 1985 ). Naturalistic inquiry . Newbury Park, CA: Sage Publications.

Lincoln, Y. ( 1995 ). ‘ Emerging Criteria for Quality in Qualitative Research and Interpretive Research ’, Qualitative Inquiry , 1 (3), 275–289.

Lincoln, Y. , & Guba, E. ( 2000 ). Paradigmatic controversies, contradictions, and emerging confluences. In N. Denzin & Y. Lincoln (Eds.), Handbook of qualitative research (2nd ed.) (pp. 163–188). Thousand Oaks, CA: Sage Publications.

Lincoln, Y. , & Guba, E. ( 2005 ). Paradigmatic controversies, contradictions, and emerging confluences. In N. Denzin & Y. Lincoln (Eds.), The SAGE Handbook of qualitative research (3rd ed.) (pp. 191–216). Thousand Oaks, CA: Sage Publications.

Lincoln, Y. , Lynham, S. , & Guba, E. ( 2011 ). Paradigmatic controversies, contradictions, and emerging confluences, revisited. In N. Denzin & Y. Lincoln (Eds.), The SAGE Handbook of qualitative research (4th ed.) (pp. 97–128). Thousand Oaks, CA: Sage Publications.

Patton, M. ( 1990 ). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage Publications.

Patton, M. ( 2002 ). Qualitative research and evaluation methods (3rd ed.). Thousands Oak, CA: Sage Publications.

Popay, J. , Rogers, A. , & Williams, G. ( 1998 ). Rationale and standards for the systematic review of qualitative literature in health services research.   Qualitative Health Research , 8 (3), 341–351.

Richardson, L. ( 2000 ). Writing: A method of inquiry. In N. Denzin & Y. Lincoln (Eds.), Handbook of qualitative research (2nd ed.) (pp. 923–948). Thousand Oaks, CA: Sage Publications.

Richardson, L. , & St. Pierre, E. A. ( 2005 ). Writing: A method of inquiry. In N. Denzin & Y. Lincoln (Eds.), The SAGE Handbook of qualitative research (3rd ed.) (pp. 959–978). Thousand Oaks, CA: Sage Publications.

Riessman, C. ( 2008 ). Narrative methods for the human sciences . LA: Sage Publications.

Sale, J. , & Brazil, K. ( 2004 ). A strategy to identify critical appraisal criteria for primary mixed-method studies.   Quality & Quantity , 38 , 351–365.

Sale, J. , Lohfeld, L. , & Brazil, K. ( 2002 ). Revisiting the quantitative-qualitative debate: Implications for mixed-methods research.   Quality and Quantity , 36 , 42–53.

Scheurich, J. J. ( 1996 ). The masks of validity: A deconstructive investigation.   International Journal of Qualitative Studies in Education , 9 (1), 49–60.

Schwandt, T. ( 2002 ). Evaluation Practice Reconsidered . New York: Peter Lang.

Seale, C. ( 1999 ). The quality of qualitative research . London: Sage Publications.

Seale, C. ( 2002 ). Qualitative issues in qualitative inquiry.   Qualitative Social Work , 1 (1), 97–110.

Smith, J. ( 1990 ). Alternative research paradigms and the problem of criteria. In E. Guba (Ed.), The paradigm dialog (pp. 167–187). Newbury, CA: Sage Publications.

Stiles, W. ( 1999 ). Evaluating qualitative research.   Evidence-based Mental Health , 2 (4), 99–101.

Tashakkori, A. , & Teddlie, C. ( 2003 ). The past and future of mixed methods research: From data triangulation to mixed model designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 671–702). Thousand Oaks, CA : Sage Publications .

Tashakkori, A. , & Teddlie, C. ( 2008 ). Quality of inference in mixed methods research: Calling for an integrative framework. In M. Bergman (Ed.), Advances in mixed research: Theories and applications (pp. 101–119). London: Sage Publications.

Thomas, E. , & Magilvy, J. ( 2011 ). Qualitative rigor or research validity in qualitative research.   Journal of Specialists in Pediatric Nursing , 16 , 151–155.

Tracy, S. ( 2010 ). Qualitative quality: Eight “Big-tent” criteria for excellent qualitative research.   Qualitative Inquiry , 16 (10), 837–851.

Yin, R. ( 1999 ). Enhancing the quality of case studies in health services research.   Health Services Research , 345 (5), 1209–1224.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

An Introduction to Qualitative Evaluation Approaches

• First Online: 10 February 2022

Cite this chapter

• Charles P. Friedman 4 ,
• Jeremy C. Wyatt 5 &
• Joan S. Ash 6  

Part of the book series: Health Informatics ((HI))

1438 Accesses

This chapter is the first of three devoted to describing qualitative methods for evaluating informatics interventions. Its major goals are to describe when qualitative methods are appropriate and to offer a general framework for understanding how studies using these methods are conducted. Chapters 15 and 16 provide much more detailed tours through the methods of qualitative evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Berg BL, Lune H. Qualitative research methods for social scientists. 8th ed. Pearson: Boston, MA; 2012.

Google Scholar  

Crabtree BF, Miller WL. Doing qualitative research. 2nd ed. Thousand Oaks, CA: Sage; 1999.

Cresswell JW. Research design: qualitative, quantitative, and mixed methods approaches. Thousand Oaks, CA: Sage; 2019.

Davis WS. Business systems design and analysis. Belmont, CA: Wadsworth; 1994.

Forsythe DE. Using ethnography to build a working system: rethinking basic design assumptions. Symp Comput Applications Med Care. 1992;16:505–9.

Forsythe DE. New bottles, old wine: hidden cultural assumptions in a computerized explanation system for migraine sufferers. Med Anthropol Q. 1996;10:551–74.

Article   CAS   Google Scholar  

Forsythe DE. Using ethnography to investigate life scientists’ information needs. Bull Med Libr Assoc. 1998;86(3):402–9.

Goedhart NS, Zuiderent-Jerak T, Woudstra J, Broerse JEW, Betten AW, Dedding C. Persistent inequitable design and implementation of patient portals for users at the margins. J Am Med Inform Assoc. 2021;28(2):276–83.

Article   Google Scholar  

House ER. Evaluating with validity. Beverly Hills: Sage; 1980.

J Am Med Inform Assoc 2021;28(2):197–423. (special issue).

Kaplan B. Initial impact of a clinical laboratory computer system. J Med Syst. 1987;11:137–47.

Kaplan B, Duchon D. Combining qualitative and quantitative methods in information systems research: a case study. MIS Q. 1988;12:571–86.

Keikhosrokiani P. Perspectives in the development of mobile medical information systems: life cycle, management, methodological approach and application. Amsterdam: Elsevier; 2019.

Norman DA. The design of everyday things. New York: Basic Books; 1996.

Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Heath Care. 2010;19(suppl 3):i68–74.

Strauss A, Corbin J. Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, CA: Sage; 1990.

Toye F. Not everything that can be counted counts, and not everything that counts can be counted. Br J Pain. 2015;9:7.

University of Technology Sydney. Adapting research methods in the COVID-19 pandemic: resources for researchers, 2nd ed. December, 2020.

Winograd T, Flores F. Understanding computers and cognition: a new foundation for design. Reading, MA: Addison-Wesley; 1987.

Download references

Author information

Authors and affiliations.

Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, USA

Charles P. Friedman

Department of Primary Care, Population Sciences and Medical Education, School of Medicine, University of Southampton, Southampton, UK

Jeremy C. Wyatt

Department of Medical Informatics and Clinical Epidemiology, School of Medicine, Oregon Health & Science University, Portland, OR, USA

Joan S. Ash

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Charles P. Friedman .

Answers to Self-Tests

Self-test 14.1.

Qualitative methods can be especially useful at the beginning of a project because they can yield information in the form of a needs assessment. Later, after implementation, they can be extremely valuable if used to provide feedback for improving the information resource.

Systems analysis processes attempt to be sensitive to the context into which the system will be placed and they are also iterative like qualitative processes.

These methods are inductive and interpretive because they require the study team to be open minded, to learn from participants, and to offer explanations and describe the meaning of cultural attributes.

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Friedman, C.P., Wyatt, J.C., Ash, J.S. (2022). An Introduction to Qualitative Evaluation Approaches. In: Evaluation Methods in Biomedical and Health Informatics. Health Informatics. Springer, Cham. https://doi.org/10.1007/978-3-030-86453-8_14

Download citation

DOI : https://doi.org/10.1007/978-3-030-86453-8_14

Published : 10 February 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-86452-1

Online ISBN : 978-3-030-86453-8

eBook Packages : Medicine Medicine (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Technical Support
• Find My Rep

You are here

How to Use Qualitative Methods in Evaluation

• Michael Quinn Patton - Utilization-Focused Evaluation, Saint Paul, MN
• Description

See what’s new to this edition by selecting the Features tab on this page. Should you need additional information or have questions regarding the HEOA information provided for this title, including what is new to this edition, please email [email protected] . Please include your name, contact information, and the name of the title for which you would like more information. For information on the HEOA, please go to http://ed.gov/policy/highered/leg/hea08/index.html .

For assistance with your order: Please email us at [email protected] or connect with your SAGE representative.

SAGE 2455 Teller Road Thousand Oaks, CA 91320 www.sagepub.com

A very good book introducing students to qualitative methodology

Text provided pertinent information that facilitated a clear understanding of the use of qualitative methods in evaluation.

Michael Patton's ' How to use qualitative methods in evalution', written with pedagogic clarity, is bound to become one of the must read in evaluation studies

This is the core textbook for my module Evaluating Educational Research & Practice. It is packed full of exercises which are intended to deepen students knowledge & awareness of the issues included in the textbook.

For instructors

Select a purchasing option.

• make a call
• schedule an appointment
• Chat Loading...
• Request Info

855-725-7614

• Online Graduate Degree Programs
• School of Public Affairs
• Master of Public Administration and Policy
• MS in Counter-Terrorism and Homeland Security
• College of Arts and Sciences
• MA in Economics, Applied Economics Specialization
• MS in Nutrition Education
• School of Communication
• School of Professional and Extended Studies
• MS in Health Promotion Management
• MS in Human Resource Analytics and Management
• MS in Measurement & Evaluation
• MS in Sports Analytics and Management
• Online Graduate Certificates
• School of Professional and Extended Studies Graduate Certificate Programs
• Graduate Certificate in Human Resource Analytics and Management
• Graduate Certificate in Nutrition Education
• Graduate Certificate in Project Monitoring & Evaluation
• Graduate Certificate in Sports Analytics and Management
• Financial Aid and Tuition
• Scholarships
• International Students
• Military Students
• Schedule an Appointment
• University Registrar
• Campus Programs
• Why American University
• Accreditation and Rankings
• President's Message
• Online Student Life
• Virtual Open Houses & Webinars
• Frequently Asked Questions

Qualitative Methods in Monitoring and Evaluation: Thoughts Considering the Project Cycle

• Online Degrees
• Program Resources

You are here

As we monitor and evaluate projects, we use many different kinds of qualitative methods, and each of these methods gives us different kinds of data.  Depending on our evaluation statement of work or performance monitoring plan, we use different methods on particular occasions to elicit certain kinds of data.

As we craft our qualitative or mixed method evaluation designs, we should consider what qualitative methods we would use, and what kind of data those methods would give us.  Evaluators have a large toolkit of qualitative methods, and we use each of these methods under different circumstances to gather different kinds of data.  As Nightengale and Rossman (2010) explain, we need to decide what our unit of analysis will be; the number of sites that we will use; how we will choose those sites; what data we need; and what method will give us that data.  We also need to consider Bamberger, Rugh, and Mabry's (2012) constraints of time, budget, data, and politics as we plan our qualitative research and evaluations. We should pay special attention to ethical considerations, as qualitative researchers tend to spend a lot of time with informants, gathering sensitive data in the process.

Let’s consider the use of several qualitative methods through the project cycle, from planning, to implementation, and project conclusion.  We should consider what qualitative methods we would use, and what kind of data those methods would give us. 

Planning As we are planning our project, if we are lucky, a donor will give us money to carry out a needs assessment.   A quantitative needs assessment, perhaps even using already existing data, might tell us literacy rates or hospitalization rates, for example.  This kind of data can be important for our project, depending on its scope, objectives, and activities. 

A qualitative needs assessment might give us more of a disaggregated perspective of literacy or health issues that takes into account emic perspectives.  Observation might give us a picture of what is happening in the project setting.  Participant observation might give us more of an emic understanding of what is happening, especially if we are allowed into the backstage where the observer effect is no longer as evident.  At this stage, key informant interviews might give us some possible project parameters, and this might be of particular importance if there are gatekeepers in the community who could help or hinder a project and its activities.  Participatory tools like seasonal calendars might help us to understand the emic needs of the community, and the different local events or micropolitics issues that might impact project implementation and beneficiary access. 

Understanding the needs of the community is an important process, and with emic data we can construct projects and activities and set indicators that are culturally appropriate. 

Another aspect here is baseline data collection .  We sometimes collect this as we are planning our project, and we sometimes collect it just before we start our activities.  Collecting baseline data may be important if we want to be able to show outcomes or conduct an impact assessment after the conclusion of our project.  If we want to show the impact of our project, or the changes in people’s attitudes, behaviors, or competencies, then we may need a baseline to compare to.  Depending on our project, we might use a census table or a structured interview schedule to collect baseline data during the planning phase of a project.

Implementation We incorporate qualitative data into our monitoring efforts and formative evaluations so that we can improve project activities. We adapt and learn from our project’s implementation when we carry out formative evaluations. 

Qualitative methods that monitor progress are particularly important during the implementation phase of a project.  Using qualitative data to monitor projects gives us insight into a project’s activities as they are being implemented.  This can be more helpful to us than quantitative data, such as “number of people trained.”  Indeed, one of the most common uses of qualitative data is to help explain or add perspective to quantitative data.  We can use qualitative data to tweak or change direction of our programming, especially if we are not hitting our intended objectives or making progress towards our indicators.

We use observation to see what is happening in our project, who is participating, and who is not participating.  We use participant observation and key informant interviews to understand what is happening in our project as it is being implemented.   Focus groups and participatory tools are also important for us so that we can get a wider perspective of project activities and outputs.  

Outcomes and Impact Showing causation between the baseline and outcome data is something to consider in the design of an impact evaluation. Without that baseline data, we might not be in a position to show our project’s impact, so we need to think about collecting baseline data during the planning or at the start of the implementation phase if we want to show this later on.

As above, observation and participant observation allow us to observe and understand change that has or has not taken place in society as a result of our program.  Key informant interviews and focus groups give us insight into the change, or lack thereof.

Concluding Thoughts While our evaluation designs need to be solid, we also need to have knowledge to implement the designs within other particular historical, cultural, and linguistic settings.  Our designs are only going to take us so far, and that we as evaluators need training and expertise to use qualitative methods in culturally appropriate ways.

References: Michael Bamberger, Jim Rugh, and Linda Mabry, Real World Evaluation: Working Under Budget, Time, Data, and Political Constraints, Thousand Oaks: SAGE, 2012. Demetra Smith Nightengale and Shelli Rossman, “Collecting Data in the Field,” in Joseph Wholey, Harry Hatry, and Kathryn Newcomer, eds., Handbook of Practical Program Evaluation, San Francisco: Wiley, 2010.

About the Author: Dr. Beverly Peters has more than twenty years of experience teaching, conducting qualitative research, and managing community development, microcredit, infrastructure, and democratization projects in several countries in Africa. As a consultant, Dr. Peters worked on EU and USAID funded infrastructure, education, and microcredit projects in South Africa and Mozambique. She also conceptualized and developed the proposal for Darfur Peace and Development Organization’s women’s crisis center, a center that provides physical and economic assistance to women survivors of violence in the IDP camps in Darfur. Dr. Peters has a Ph.D. from the University of Pittsburgh. Learn more about Dr. Peters .

To learn more about American University’s online MS in Measurement & Evaluation or Graduate Certificate in Project Monitoring & Evaluation,  request more information  or call us toll free at 855-725-7614.

(855) 725-7614

• Terms & Conditions
• About American University

• Open access
• Published: 01 May 2024

Evaluation and students’ perception of a health equity education program in physical therapy: a mixed methods pilot study

• Alexis A. Wright 1 ,
• Dominique Reynolds 1 &
• Megan Donaldson 2  

BMC Medical Education volume  24 , Article number:  481 ( 2024 ) Cite this article

Metrics details

Health equity is a common theme discussed in health professions education, yet only some researchers have addressed it in entry-level education.

The purpose of this study is to serve as an educational intervention pilot to 1) evaluate students’ perception of the effectiveness of the DPT program in providing a foundation for health equity education, with or without the benefit of a supplemental resource and 2) establishing priorities for the program related to educating students on health inequities in physical therapy clinical practice.

A mixed method design with a focus-group interview was utilized to explore students’ perceptions of the DPT program's commitment to advancing health equity.

A three-staged sequential mixed methods study was conducted. Stage 1 began with quantitative data collection after completing the DEI Bundle utilizing the Tripod DEI survey. Stage 2 involved identifying themes from the Tripod Survey data and creating semi-structured interview questions. Stage 3 consisted of a focus group interview process.

A total of 78 students completed the Tripod DEI survey upon completing 70% of the curriculum. Thirty-five students, eight core faculty, 13 associated faculty, and four clinical instructors completed the APTA DEI Bundle Course Series. According to the Tripod DEI Survey results, program stakeholders found the program’s commitment to DEI and overall climate to be inclusive, fair, caring, safe, welcoming, and understanding of individuals from different backgrounds, including a sense of student belonging where students feel valued and respected. Three themes emerged from the qualitative focus group interviews, including the value of inclusivity, health equity curricular foundations, and DEI in entry-level DPT education.

Conclusions

This study highlights the value of incorporating health equity and DEI topics into curricula while fostering an incluse program culture.

Peer Review reports

Introduction

Racial and ethnic disparities in healthcare are a longstanding and well-documented crisis in the United States [ 1 ]. A strategic goal of the American Physical Therapy Association (APTA) is to increase diversity, equity, and inclusion within the profession to serve society's health better. At its core, physical therapy is rooted in optimizing overall health and decreasing preventable illness and injury. Additionally, physical therapists are trained to be adaptive and respond to patients' social and environmental influences that impact health outcomes. These foundational traits uniquely position healthcare providers with the skills to respond to health inequities. Education and training for health providers are rarely studied to determine the effectiveness or implementation of the educational training [ 1 , 2 ]. Specifically, diversity, equity, and inclusion (DEI) education training provides a basis to confront systemic racism and improve health equity, and physical therapy programs are being called to action [ 2 ]. However, the measurement of learners’ awareness and perceived effectiveness of educational interventions has lagged [ 1 ].

The literature review on this topic includes a study by the Institute of Medicine (IOM), which has provided recommendations for addressing and eliminating racial/ethnic disparities in healthcare. These recommendations include increasing healthcare providers’ awareness of racial/ethnic disparities in healthcare and educating health providers on health disparities, cultural competence, and the impact of race/ethnicity on clinical decision-making [ 3 ] A developing entry-level Doctor of Physical Therapy program intentionally designed curricula aligned with the IOM recommendations. Curricular topics were informed by the Clinical Prevention and Population Health Curriculum Framework, a product of the Healthy People Curriculum Task Force established in 2002 by the Association for Prevention Teaching and Research (APTR) [ 4 ]. Knowledge-based activities were designed to further awareness and understanding of the social determinants of health, health prevention, cultural awareness, health inequities, healthcare accessibility, systems thinking, and implicit and explicit bias among entry-level DPT students. The theoretical framework of the DPT curriculum is based on a theoretical framework of constructivism, which refers to the belief that learners actively construct knowledge by linking new information to what they have previously learned and by incorporating new experiences into their knowledge base and that learners’ knowledge structures are continually constructed and reconstructed [ 5 ].

Additionally, co-curricular educational activities were promoted throughout the program.

The theoretical framework for co-curricular educational activities is based on relational learning. Specifically, this model has been used for health promotion and inclusion [ 6 , 7 ]. The co-curriculum does what the standard academic curriculum generally does not: it is developmental, transformative, and future-focused. For example, as a program, sessions were provided for learners to attend speaker sessions on DEI topics, apply for leadership roles (including the Diversity, Equity, and Anti-Racism (DEAR) Council), and engage in service activities, all grounded in an expectation of professional behaviors that encourage intellectual discussions on complex topics in an environment free of criticism, discrimination, harassment or any other emotional or physical harm.

The purpose of this study is to serve as an educational intervention pilot to 1) evaluate students’ perceptions of the effectiveness of the DPT program in providing a foundation for health equity education, with or without the benefit of a supplemental resource, and 2) establish priorities for the program related to educating students on health inequities in physical therapy clinical practice.

Materials and methods

Participants and study design.

Determining the research question(s) is vital in the mixed research process. Research questions are pivotal in the mixed research process, which is interactive, emergent, fluid, and evolving [ 8 ]. As Leech and Onwuegbuzie [ 8 ] defined, “mixed methods research questions combine or mix both the quantitative and qualitative research questions necessitating the resulting data be collected and analyzed.” Mixed research sampling designs can be classified according to (a) the time orientation of the components (e.g., whether the qualitative and quantitative phases occur concurrently or sequentially) and (b) the relationship of the qualitative and quantitative samples (e.g., identical vs. parallel vs. nested vs. multilevel).

Design:  To address the objectives of this study, a partially mixed-method design with a sequential and nested relationship was selected. The nested structure implies that individuals chosen for one phase of the study (qualitative focus group interviews) constitute a subset of those selected in the preceding phase (participants in the quantitative surveys) [ 8 , 9 ]. Nonetheless, qualitative and quantitative research methodologies hold equal significance in this study's design and analytical approach.

Sampling Strategy: Participant enrichment refers to the mixing of qualitative and quantitative techniques for the rationale of optimizing the sample. Beginning with Phase 1, a total of 153 participants, including students (81) from the Class of 2022 (as pre-professionals) and 2) program faculty (16), associated faculty (36), and clinical instructors (20) (as post-professionals) were offered the option to participate in this mixed methods study. An email describing the purpose of the study was sent to all participants.

Within mixed-method designs, instrument fidelity is essential and used by researchers to maximize the appropriateness and utility of the quantitative and qualitative instruments used in the study. These included the Tripod DEI survey, the APTA Diversity, Equity, and Inclusion (DEI) Bundle, and the qualitative semi-guided interview process. Stage 1 began with quantitative data collection after completing the Diversity, Equity, and Inclusion Bundle utilizing the Tripod DEI survey. Stage 2 involved identifying themes from the Tripod Survey data and creating semi-structured interview questions. Stage 3 consisted of the focus group interview process. See further details outlining the timeline and phases of the study in Fig.  1 . Timeline and Process for Study.

Timeline and Process for Study

The research implementation began with the quantitative survey, in which all students were surveyed using the Tripod DEI survey, which was deployed after semester 4 of the program, reflecting 70% completion of the curriculum [ 10 ]. Students were allowed to participate in the voluntary, supplementary APTA DEI Bundle beginning in Semester 5 [ 11 ]. Before participating in the APTA DEI Bundle, the Tripod DEI Survey was readministered to all students, program faculty, associated faculty, and clinical instructors who elected to participate [ 10 ]. Following completion of the APTA DEI Bundle, the Tripod DEI Survey was readministered a second time to all students, program faculty, associated faculty, and clinical instructors who completed the APTA DEI Bundle course series [ 10 , 11 ]. The pre-test and post-test methodologies explored differences between adding the American Physical Therapy Association DEI Bundle to the program’s curriculum and co-curricular activities [ 11 ].

The study commenced once approval to conduct it was obtained from the Institutional Review Board at the university. After the submission was reviewed, the Tufts University IRB office determined that the proposed activity was not deemed human research as defined by DHHS and FDA regulations. (IRB ID:STUDY00002820).

Research planning: quantitative study instrument

Tripod Education Partners works with programs to gather, organize, and report on student and teacher perspectives [ 10 ]. The Tripod DEI survey captures student perceptions of how diversity, equity, and inclusion issues play out in their school. The survey collects feedback from teachers about their experiences as teachers and perspectives about strengths and opportunities for improvement. Permission and funding for survey distribution were obtained before disseminating the survey.

The survey consisted of a total of 38 questions with eight distinct measures including 1) School commitment to DEI ( N  = 3), 2) School climate overall ( N  = 4), 3) School climate for DEI ( N  = 4), 4) Classroom teaching supporting DEI ( N  = 7), 5) Co-Curricular activities supporting DEI ( N  = 3), 6) Everyday discrimination by students ( N  = 6), 7) Everyday discrimination by teachers ( N  = 6), 8) Meaningful interactions across difference N  = 5) (Tripod Education Partners,2019). School commitment to DEI is scored on a Likert scale from 1 (totally untrue) to 5 (totally true). School climate overall and DEI are scored as ordinal variables, with 2 being more favorable. Classroom teaching supporting DEI is scored on a Likert scale from 1 (none) to 5 (all). Co-curricular activities supporting DEI is scored on a Likert scale from 1 (my school doesn’t sponsor things like this) to 6 (very often). Everyday discrimination by students and teachers and meaningful interactions across differences are scored on a Likert scale from 1 (never) to 5 (very often).

The “overall sense of belonging” ( N  = 3) was scored on a Likert scale from 1 (totally untrue) to 5 (totally true).

The Tripod DEI survey development shows good construct validity and internal consistency [ 10 ]. Diverse student populations are at the center of the survey. Reports disaggregate findings by social identities across various groups, including but not limited to race, gender, and socioeconomic status. This breakdown allows programs to pinpoint groups of students reporting less-than-positive experiences and take action to address their needs.

Research planning: description of the DEI training bundle

The optional training program was conducted through asynchronous electronic delivery of the APTA DEI bundle [ 11 ]. This program is a three-part series exploring foundational concepts related to diversity, equity, and inclusion and is led by Diana Lautenberger, MA, co-lead of the American Medical Colleges' leadership development seminar program. The three-part series utilizes a highly reflective approach whereby participants learn about identity, privilege, bias, and allyship as foundational pillars to achieving DEI. In addition, participants engage in self-reflection throughout the series to apply concepts to their clinical and personal lives to create more respectful and inclusive environments.

The series consists of three two-hour sessions: Part 1 – Unconscious Bias in the Health Professions; Part 2 – Power, Privilege, and Microaggressions; Part 3 – Responding to Microaggressions Through Allyship. The elements of this bundle listed objectives for the learners to 1) understand how their various identities carry social capital or power, 2) describe aspects of a dominant culture that advantage some and disadvantage others, and 3) utilize allyship and bystander intervention strategies that reduce harm to create more respectful and inclusive environments [ 11 ]. This program requires the completion of an assessment from the training. Viewers who completed all three sessions and scored at least 70% on each session's assessment (built into the modules) were also allowed to earn 0.6 CEUs (six contact hours) and a certificate of completion.

Research planning: qualitative focus group interviews

Using an explanatory sequential mixed methods study, the qualitative portion aimed to further understand the students’ perceptions, establish priorities for the program related to educating students on health inequities in physical therapy clinical practice, and evaluate the effectiveness of adding the DEI Bundle. Based on the results of the quantitative portion of the study, two researchers created questions that would be used in the focus group interviews. The a priori semi-structured question guide in Table  1 was designed to allow emergent focus group discussion to explore concepts further.

Data analysis plan

Quantitative data collection and analysis.

The data analysis program IBM SPSS 28.0 was utilized to store and analyze data from the Tripod DEI survey. For all the Tripod DEI survey subscales, items were summed, and scores were calculated. Descriptive statistics were utilized to calculate means, standard deviations, and 95% confidence intervals for each of the eight domains and Overall Sense of Belonging. Paired sample t-tests were conducted to compare pre-test and post-test scores. Summary independent samples t-tests compared the entire sample data ( N  = 81) to the post-DEI Bundle Series data.

Qualitative data collection and analysis

The semi-structured focus group interview guide questions (Table  1 ) were designed after the quantitative data collection was completed, and the data assessment revolved around concepts collected from the survey data.

A variety of data collection strategies were used, including (a) a mixture of open- and closed-ended items within the questionnaires that guided the focus group interview process, (b) a mixture of a priori (from the quantitative results) and additional emergent/flowing focus-group strategies through a semi-guided interview process. The Standards for Reporting Qualitative Research (SRQR) checklist was utilized for reporting.

Given the small sample size, no statistical software was utilized. Coding was used to assign labels to data segments to capture their meaning and allow comparison to identify themes or patterns. Both researchers used qualitative content analysis to systematically categorize transcribed content into topic areas from the thick descriptions provided. Qualitative fields were created to organize data by topic counts of language content areas (such as “DEI” and “belonging” quotes). The preliminary or open coding was done first and then refined to a higher level to reflect broader categories. All coding stages were done separately and then together to ensure improved accuracy. Then, the researchers used the comparison analysis and consensus approach to categorize and interpret data to identify patterns and content themes during the analysis. The analysis used a matrix table as a visual spreadsheet, where the rows represented participants, and the columns represented codes identified.

Researcher characteristics and reflexivity: The background and experience of the researchers could have influenced the research as two of the researchers had routine involvement with the participants within the study. The same researchers that conducted the study design and implementation conducted the focus group interviews via Zoom while participants were on clinical rotations. The focus-group interviews were audio-recorded and transcribed by an administrative coordinator who supported the faculty and had limited student interactions during daily work.

Techniques to enhance trustworthiness: The research team, consistent throughout the study, undertook the quantitative and qualitative data analysis. To maintain objectivity, they devised a set of a priori questions for interviews, steering clear of leading inquiries or interpretations. Subsequently, they conducted content analysis directly from transcriptions. Reflexivity strategies encompassed credibility checks via member validation and a post-session peer debriefing (between researchers), ensuring accuracy in focus group interviews. The research coordinator, unbiased to quantitative analysis, remained uninvolved in question formulation, solely providing session transcriptions for analysis. Furthermore, thick descriptions were provided, and qualitative counts of language content areas were evenly applied to promote the transferability of qualitative findings. By integrating these measures, the study aimed to mitigate inherent limitations in its design and bolster the credibility, transferability, dependability, and confirmability of its qualitative research, thus enhancing the trustworthiness and reliability of its findings.

Quantitative analysis and results

A total of 78 students completed the Tripod DEI survey upon completing Semester 4 of the curriculum. A total of 42 students, eight core faculty, 16 associated faculty, and four clinical instructors elected to participate and complete the voluntary, supplementary pre-APTA DEI bundle Tripod DEI survey beginning Semester five. A total of 35 students, eight core faculty, 13 associated faculty, and four clinical instructors completed the APTA DEI Bundle Course Series. Thirty-two students, eight core faculty, 13 associated faculty, and four clinical instructors completed the post-APTA DEI Bundle Tripod DEI Survey.

Student results

Demographics of the full sample of 78 students can be found in Table  2 .

Survey results following the completion of Semester 4 are summarized below and reported as mean, standard deviation.

School Commitment to DEI (1 = totally untrue; to 5 = totally true)

Students generally found the program's commitment to DEI to be inclusive, fair, and understanding of individuals from different backgrounds (M = 4.1, SD = 0.9) or “mostly true”.

School Climate Overall (1 = less favorable; 2 = favorable)

Students reported the program's climate/culture as caring, respectful, safe, and welcoming (M = 2.0, SD = 0.1) where 2 is scored as caring, respectful, safe, and welcoming.

School Climate for DEI (1 = less favorable; 2 = favorable)

Students rated the program's climate/culture for DEI as “equally fair” to all students, regardless of their social identity (M = 1.9, SD = 0.2). This included questions related to race, ethnicity, sexual orientation, socioeconomic status, and gender where 2 is scored as equally fair to all students.

Classroom teaching Supporting DEI (1 = none; 5 = all)

Classroom teaching supporting diversity, equity, and inclusion rated “most but not all” (M = 4.1, SD = 0.8) faculty as having integrated material on different social identities, discussing issues of social inequality, and using student-centered teaching methods. This included questions related to helping students think about how to improve the world, leading discussions about why some people have difficult lives and other people have easier lives, connecting content from the classroom to problems or issues in the world as well as the student’s own life and interests, helping students think about how to improve other people’s lives, assigning readings or materials about people from different backgrounds or places, and taught about influential people from many different cultures.

Co-Curricular Activities Supporting DEI (1 = my school doesn’t sponsor things like this; 6 = very often)

With regards to co-curricular activities supporting diversity, equity, and inclusion, students reported on average that they “hardly ever” participated in a school-sponsored group for students of different racial, ethnic, socioeconomic, gender, sexual orientation, or ability groups; attended a school-sponsored event related to diversity, fairness, or inclusion; or participated in a program sponsored group working to make the world a better place (M = 3.3, SD = 1.0).

Everyday Discrimination by Students (1 = never; 5 = very often)

Students reported “never to hardly ever” regarding everyday discrimination by students regarding courtesy, respect, intelligence, being better than others, being bullied or threatened, and insults (M = 1.8, SD = 0.7).

Everyday Discrimination by Teachers (1 = never; 5 = very often)

Students reported “never to hardly ever” regarding everyday discrimination by faculty regarding courtesy, respect, intelligence, being better than others, being bullied or threatened, and insults (M = 1.4, SD = 0.6).

Meaningful Interactions Across Differences (1 = never; 5 = very often)

Students rated the program as “fairly often” with regards to meaningful interactions across differences, including honest discussions with other students whose religion was different from their own, whose families have more or less money than their own, whose culture is different from their own, and whose race is different from their own (M = 3.8, SD = 0.9).

Belonging (1 = totally untrue; 5 = totally true)

Finally, the students rated the program as “mostly true to totally true” concerning their sense of belonging in the program, whereby the student feels valued, respected, and a sense of belonging (M = 4.4, SD = 0.8).

Comparison of tripod survey pre-post

Thirty-two students elected to participate and complete the APTA DEI Bundle Series with completed pre- and post-Bundle Series survey data. Demographic information on student participation in the DEI Bundle can be found in Table  3 . After completing the APTA DEI Bundle Series, we found no significant difference in any of the eight domains or Sense of Belonging. We found no significant differences in any domain between the full sample ( N  = 78) and the post-DEI Bundle Series data sample ( N  = 32).

Post-professional stakeholder results

Twenty-five of our post-professional stakeholders elected to participate and complete the APTA DEI Bundle Series with completed pre- and post-Bundle Series survey data. After completing the APTA DEI Bundle Series, we found no significant difference in any of the eight domains or Sense of Belonging.

Similarly, the post-professional stakeholders generally found the program's commitment to DEI to be inclusive, fair, and understanding of individuals from different backgrounds (M = 4.2, SD = 1.2).

Post-professionals reported the program’s climate/culture overall as caring, respectful, safe, and welcoming (M = 2.0, SD = 0.0).

Post-professionals rated the program’s climate/culture for DEI as “equally fair” to all students, regardless of their social identity (M = 2.0, SD = 0.1). This included questions related to race, ethnicity, sexual orientation, socioeconomic status, and gender.

Classroom Teaching Supporting DEI (1 = none; 5 = all)

Post-professionals rated climate for DEI Classroom teaching supporting diversity, equity, and inclusion rated “most but not all faculty” (M = 3.8, SD = 1.0) as having integrated material on different social identities, discussing issues of social inequality, and using student-centered teaching methods. This included questions related to helping them think about how to improve the world, leading discussions about why some people have difficult lives and other people have easier lives, connecting content from the classroom to problems or issues in the world as well as the student’s own life and interests, helping students think about how to improve other people’s lives, assigning readings or materials about people from different backgrounds or places, and taught about influential people from many different cultures.

With regards to co-curricular activities supporting diversity, equity, and inclusion, post professionals reported on average that they “hardly ever participated” in a school-sponsored group for students of different racial, ethnic, socioeconomic, gender, sexual orientation, or ability groups; attended a school-sponsored event related to diversity, fairness, or inclusion; or participated in a program sponsored group working to make the world a better place (M = 2.9, SD = 1.0).

Post professionals reported “never to hardly ever” concerning everyday discrimination by students (M = 1.3, SD = 0.5).

Post professionals reported “never to hardly ever” concerning everyday discrimination by teachers (M = 1.4, SD = 0.5).

Post professionals rated the program as “fairly often” with regards to meaningful interactions across differences, including honest discussions with other students whose religion was different from their own, whose families have more or less money than their own, whose culture is different from their own, and whose race is different from their own (M = 3.1, SD = 0.9).

Finally, the post professionals rated the program as “mostly true to totally true” regarding their sense of belonging in the program, whereby the student feels valued, respected, and a sense of belonging (M = 4.5, SD = 1.0).

Result of qualitative focus group content analysis

From those participants completing the quantitative portion of the study, a nested sub-group of students ( n  = 9) volunteered to participate in the semi-structured focus group interview following the completion of the DEI Bundle. Demographic information on student participation in the interviews can be found in Table  4 .

There was a rich discussion with the interview guide around the topics 1) DEI with or without the training supplement related to health equity in physical therapy and 2) the program’s commitment to training students on topics associated with health equity. Three themes emerged from the qualitative focus group interviews based on the final qualitative content analysis.

Theme 1: student’s perceived value of inclusivity

Theme one was the value of inclusivity with three associated sub-themes of fairness, actions, and communication. In higher education, inclusivity is the ongoing process of improving the education system to meet the needs of all students, especially those in marginalized groups. Inclusivity involves reimagining educational services to cater to a diverse audience and making learning materials and teaching methods accessible to as many students as possible. This includes considering a range of diverse student identities, including race, gender, sexuality, and abilities. “ The program does make an effort, especially with adjuncts that we bring in, ableism talks, and people from different backgrounds speaking to us in classes on Zoom .”

Additionally, providing sessions to improve inclusivity and communicating and demonstrating actions consistent with the value of inclusivity is essential to the participants. “ Being a member of the gay community, having a faculty in class that you feel you belong in and are not outcasted in is super important .” Participants valued being included during activities and communicating support during school and personal life challenges. The participants recognized the challenge of finding people from different backgrounds who meet the expectations and specialties to teach within the program. They identified that, at times, visual diversity was limited within the core faculty but felt an intention of more inclusivity of race and ethnicity within the associated faculty roles or lecturers.

Within the value of inclusivity, there is also an inherent limitation to who can afford the DPT graduate-level program at a private university. Hybrid education offers more geographical convenience and reaches a more diverse student group; however, current students feel that money concerns could be a barrier to inclusivity, especially those in marginalized groups. “Program doesn’t have control over the cost of tuition but does communicate what is available as far as opportunities for financial aid.” However, they felt that communication about costs for the hybrid program and what financial aid was available was essential.

Theme 2: student’s perceived value of health equity curricular foundations

Theme two was the value of health equity curricular foundations with three sub-themes of representation in assignments, system resources, and practice issues. Health equity is the goal of helping people reach their highest level of health. It means everyone has a fair chance to achieve optimal health regardless of race, ethnicity, gender identity, or socioeconomic status. Health equity can be promoted through DEI initiatives, which focus on representing the acceptance and inclusiveness of people. The focus group reported health equity topics associated with race, social determinants, and access were satisfactorily addressed within the curriculum. However, there were opportunities to gain additional insights on improving formative activities to be more integrated with how health issues affect those with visual diversity. “ Activities within the program should also include skin tone other than white throughout systems-focused curriculum case studies, mannequins, and simulation/ standardized patients .”

Theme 3: student’s perceive value of DEI in entry-level PT education

Lastly, one remaining theme specifically addressed DEI supplementation to the curriculum. Theme three is the value of DEI in entry-level physical therapy education, with three sub-themes emerging on the timing of content, planned redundancy of learning, and the limited value of a stand-alone DEI bundle. The students in the focus group had a consensus on their perceived confidence and appropriate knowledge of social determinants of health when working with the underserved population during their clinical education exposures. However, the focus group agreed with “ concerns about generalizing their feelings to all classmates, as some students may have had different experiences based on their final clinical education setting and exposure .”

Additionally, according to the student perception, inclusivity and health equity values should be blended across the curriculum so that support and the training of those with different backgrounds can be promoted through DEI initiatives. Curriculum initiatives were given rich context regarding the program and curriculum that would be more “ inclusive and supportive of a health equity curricular track and activities threaded throughout the curriculum rather than a stand-alone module .” There was a consensus from the focus group that mirrored the quantitative results that there was a perceived “ limited value in the DEI Bundle as a stand-alone module outside of the curriculum .” Instead, the students preferred the curriculum designed to include the topics sufficiently within systems and population coursework.

The mixed methods analysis allows a better explanation of the student’s perceptions by blending the results from this study's qualitative and quantitative study portions. It was found in both portions of the study design that the program climate/culture is essential, especially as students relate inclusivity and accepting others when learning to value DEI from a health equity perspective. Students further strengthened their perceived value for their education and blended content topics across the curriculum as they related to health equity and diversity. Still, they found value when more than just content was presented. Students felt that there was a program culture, planned curriculum content, and co-curricular (outside of a class) support for health equity and inclusivity of the population's health care providers serve. As educators look to streamline variation in essential content across healthcare disciplines, utilizing a structured format (toolkit or bundle) could benefit students educationally but may be valued less by them.

Our study aimed to explore the students’ perceptions and establish priorities for the program regarding educating students on health inequities in physical therapy clinical practice.

Health equity is a common theme discussed in health professions education, yet only some have published the methods to address it in entry-level education. National organizations recommend that medical schools and health professions train students in the social determinants of health. This provides the opportunity to educate the next generation of healthcare professionals about sensitive yet essential issues.

Given the complexity of this topic, we utilized a three-staged sequential mixed methods approach to generate the results presented in this study. We found the program’s commitment to DEI and overall climate to be inclusive, fair, caring, safe, welcoming, and understanding of individuals from different backgrounds, including a sense of student belonging where students feel valued and respected. Additionally, the sample provided feedback on the educational approach and format, which was provided with the DEI Bundle. The modular-based curricular approach (not integrated through a course) was used in this study. Thus, the results of the APTA’s DEI Bundle should be considered, given the context of the study, regarding the curricular delivery and format as an “addition to” approach. Given this format, the DEI Bundle was insignificant due to the threaded curricular approach already within the program, as assessed on the Tripod DEI survey or qualitative focus group theme. This approach aligns with other recommendations for curriculum approaches to health equity [ 12 ] that integrate health equity content longitudinally and alongside other topics. The goal would be to eliminate views of health equity and healthcare as separate [ 13 ].

Limited studies explore health equity topics' style, content, and delivery through the healthcare professional’s entry-level educational program. However, the Association of American Medical Colleges recommends that medical educators expose their students to content about health disparities [ 14 ]. There are some challenges to implementing the recommendations [ 15 ], which are further complicated by the lack of recommendations regarding format, delivery, and the requisite degree of competency, which are poorly defined. Several resources are provided but not easily found across all health professions disciplines. However, several studies highlight the importance of health equity education, its impact on therapeutic relationships (trust and caring), and identify the consequences of implicit bias on patient adherence and outcomes [ 16 ].

Significant work must be done to unite all the health professions on strategies for implementing the health equity curriculum. However, an external resource strategy or modular-based approach could be effective, given limited resources and a lack of topic expertise within the program faculty. Still, it should be used with an integrated approach and placed intentionally within the curriculum design. It should have more opportunities for integration across courses, with case studies to facilitate thinking and reasoning and culminate in a competency type of assessment. Curriculum toolkits provided by professional associations may be one way to unite the disciplines to support health equity education in the health professions [ 17 ]. An excellent example of this approach is the American Academy of Family Practitioners Health Equity Curricular Toolkit, which has over 40 content experts [ 18 , 19 ]. A threaded curriculum with a program culture and willingness to utilize health equity curriculum toolkits are essential for our next generation of health practitioners. These toolkits are resources for learning and reducing the variability in education [ 18 ]. Exploring outcomes associated with toolkits may be an option to begin to explore best practices in curriculum delivery to maximize learning outcomes and competency on health equity [ 20 ]. Lastly, any health equity resource or curricular approach should facilitate the exploration of some of the most pressing questions around social determinants of health, vulnerable populations, economics, and policy from an evidence-informed perspective.

Limitations

There are several limitations that we would like to address. Within the quantitative portion of the study, the Tripod DEI survey adequately assessed overall student perception of the DPT program commitment to DEI; however, it may need more responsiveness surrounding the APTA DEI Bundle. Within any mixed methods design approach, it is important to address data fidelity during the qualitative portion. A non-investigator conducted both the survey distribution and outcome assessment; however, the focus group interviews were conducted by two study investigators. Additionally, both researchers are on the leadership team within the program, which may compromise the fidelity, trustworthiness, or sharing from the participants during this experience. It is a limitation in the study that the researchers also are involved in the education. Although a safe space and relational learning theory approach is utilized within the program, this may have limited some of the exploration of the topics/themes if the participants were sensitive. From what was shared in the focus groups, a non-investigator recorded and transcribed the data analysis portion. The second limitation of the qualitative focus groups was the limited number and need for more diversity within the sample. Specifically, the individuals who made time to participate in the qualitative focus group were not significantly diverse regarding their race or sex. The third limitation is the inability to identify the number of students who respond based on their participation in additional co-curricular activities to supplement their learning in DEI.

However, significant work must be done to unite all the health professions on strategies for implementing health equity curricula. It was essential to gain insight from the students’ perception and establish priorities on the current curriculum and entry-level education program culture related to educating students on health inequities in physical therapy clinical practice. However, given limited resources and a lack of topic expertise for health equity content among program administrators and faculty, an external resource strategy or modular-based approach could be effective. However, based on our study, the program culture is important as it relates to DEI from a health equity perspective. It should be evident to students as we influence them to become the next generation of health professionals.

Lastly, the intentional curriculum design should have more opportunities for integration across courses with case studies and culminate in a competency type of assessment, even if an external resource is used. Resources are available to support health equity education in the health professions, including health equity curriculum toolkits, which provide free links and resources for learning and may help to reduce the variability in education [ 15 ]. Any health equity resource or curricular approach should facilitate faculty’s willingness to include some of the most pressing questions around social determinants of health, vulnerable populations, economics, and policy within their current or future developed curriculum. However, motivating incremental changes in entry-level professional teaching methods and working intentionally to integrate health equity into the clinic- and classroom-based environments are tangible next steps. Identifying best practices from education to implementation has yet to be well known, and this study only provided a pilot for future studies.

Availability of data and materials

The data supporting this study's findings are available from the corresponding author upon request.

Fitzgerald C, Hurst S. Implicit bias in healthcare professionals: a systematic review. BMC Med Ethics. 2017;18:19.

Article   Google Scholar  

Matthews ND, Rowley KM, Dusing SC, Krause L, Yamaguchi N, Gordon J. Beyond a Statement of Support: Changing the Culture of Equity, Diversity, and Inclusion in Physical Therapy. Phys Ther. 2021;101(12):pzab212. https://doi.org/10.1093/ptj/pzab212 .

Institute of Medicine (US) Committee on Understanding and Eliminating Racial and Ethnic Disparities in Health Care. Unequal treatment: confronting racial and ethnic disparities in health care. In: Smedley BD, Stith AY, Nelson AR, editors. Washington (DC): National Academies Press (US); 2003. PMID: 25032386.

“Clinical Prevention and Population Health Curriculum Framework.” Association for Prevention Teaching and Research. https://www.teachpopulationhealth.org/ .

Dong H, Lio J, Sherer R, Jiang I. Some Learning Theories for Medical Educators. Med Sci Educ. 2021;31(3):1157–72. https://doi.org/10.1007/s40670-021-01270-6 . PMID:34457959; PMCID:PMC8368150.

Nieminen, J.H. Assessment for Inclusion: rethinking inclusive assessment in higher education, Teaching in Higher Education. 2022. https://doi.org/10.1080/13562517.2021.2021395 .

Rogers DT. The working alliance in teaching and learning: theoretical clarity and research implications. International Journal for the Scholarship of Teaching and Learning. 2009;3(2):Article 28.

Leech NL, Onwuegbuzie AJ. Guidelines for Conducting and Reporting Mixed Research in the Field of Counseling and Beyond. J Couns Dev. 2010;88:61–9.

Leech NL, Onwuegbuzie AJ. A typology of mixed methods research designs. Quality and Quantity: International Journal of Methodology. 2009;43:265–75.

Tripod Education Partners. Tripod’s Diversity, Equity, and Inclusion (DEI Survey) Technical Manual; 2019.

Google Scholar  

American Physical Therapy Association DEI Bundle. October 22, 2021. APTA's New DEI Course Series Helps You Work Toward More Inclusivity. 2022.  https://www.apta.org/news/2021/10/22/dei-bundle .

Butler, L., Arya, V., Nonyel, N., & Moore, T. The Rx-HEART Framework to Address Health Equity and Racism Within Pharmacy Education. American Journal of Pharmaceutical Education. 2021:85. https://doi.org/10.5688/ajpe8590 .

Landry AM. Integrating Health Equity Content Into Health Professions Education. AMA J Ethics. 2021;23(3):E229–234.

Association of American Medical Colleges. Quality improvement and patient safety competencies across the learning continuum. Association of American Medical Colleges: New and Emerging Areas in Medicine Series; 2019.

Liburd L, Hall J, Mpofu J, Williams S, Bouye K, Penman-Aguilar A. Addressing Health Equity in Public Health Practice: Frameworks, Promising Strategies, and Measurement Considerations. Annu Rev Public Health. 2020. https://doi.org/10.1146/annurev-publhealth-040119-094119 .

Paterick TE, Patel N, Tajik AJ, Chandrasekaran K. Improving health outcomes through patient education and partnerships with patients. Proc (Bayl Univ Med Cent). 2017;30(1):112–3.

Ueffing E, Tugwell P, Roberts J, Walker P, Hamel N, Welch V. Equity-oriented toolkit for health technology assessment and knowledge translation: application to scaling up of training and education for health workers. Hum Resour Health. 2009;7:67–67. https://doi.org/10.1186/1478-4491-7-67 .

American Academy of Family Practitioners. Health Equity Curricular Toolkit. Accessed 6 Jun 2023. https://www.aafp.org/family-physician/patient-care/the-everyone-project/health-equity-tools.html .

Martinez-Bianchi V, Frank B, Edgoose J, Michener L, Rodiguez M, Gottlieb L, Reddick B, Kelly C, Yu K, Davis S, Carr J, Lee JW, Smith KL, New RD. Addressing Family Medicine’s Capacity to Improve Health Equity through Collaboration. Accountability and Coalition-Building Fam Med. 2019;51(2):198–203.

Porroche-Escudero A, Popay J. The Health Inequalities Assessment Toolkit: supporting integration of equity into applied health research. J Public Health (Oxf). 2020;43:567–72. https://doi.org/10.1093/pubmed/fdaa047 .

Download references

Acknowledgements

Not applicable (NA).

This project was funded through an internal Tufts University School of Medicine Innovations in Diversity Education Awards (IDEAS) program.

Author information

Authors and affiliations.

Doctor of Physical Therapy Program, Department of Rehabilitation Sciences, Tufts University, Boston, MA, 02111, USA

Alexis A. Wright & Dominique Reynolds

Department of Rehabilitation Sciences, Medical University of South Carolina, Charleston, SC, 29425, USA

Megan Donaldson

You can also search for this author in PubMed   Google Scholar

Contributions

Authors AW and MD contributed to the work's concept, design, and data analysis; drafting and final approval of the work; and agree to be accountable for all aspects of the work. Author DR contributed to acquiring the data for the work; reviewing and final approval of the work; and agrees to be accountable for the accuracy.

Corresponding author

Correspondence to Alexis A. Wright .

Ethics declarations

Ethics approval and consent to participate.

This research study was performed following the Declaration of Helsinki. This research study was submitted to the Tufts University Social, Behavioral, & Educational Research IRB on 6/9/2022 and was determined not to meet the criteria for human research—study Protocol STUDY00002820. The IRB determined “that the proposed activity is not research as defined by DHHS and FDA regulations. IRB review and approval by this organization is not required. This determination applies only to the activities described in the IRB submission and does not apply should any changes be made.” The research team did follow all the requirements for ethical research, including the consent process. Informed consent to participate in the study was obtained from all participant data reported in this study.

Consent for publication

Competing interests, additional information, publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Wright, A.A., Reynolds, D. & Donaldson, M. Evaluation and students’ perception of a health equity education program in physical therapy: a mixed methods pilot study. BMC Med Educ 24 , 481 (2024). https://doi.org/10.1186/s12909-024-05471-6

Download citation

Received : 27 June 2023

Accepted : 25 April 2024

Published : 01 May 2024

DOI : https://doi.org/10.1186/s12909-024-05471-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Doctor of physical therapy
• Health equity
• Health professions

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 29 April 2024

The experiences and needs of older adults receiving voluntary services in Chinese nursing home organizations: a qualitative study

• Qin Shen 1 &
• Junxian Wu 1  

BMC Health Services Research volume  24 , Article number:  547 ( 2024 ) Cite this article

10 Accesses

Metrics details

Older adults living in nursing home organizations are eager to get voluntary help, however, their past experiences with voluntary services are not satisfactory enough. To better carry out voluntary services and improve the effectiveness of services, it is necessary to have a deeper understanding of the experiences and needs of older adults for voluntary services.

The purposive sampling method was used to select 14 older adults from two nursing home organizations in Hangzhou and conduct semi-structured interviews, Collaizzi’s seven-step method was used to analyze the data.

Older adults in nursing home organizations have both beneficial experiences and unpleasant service experiences in the process of receiving voluntary services; Beneficial experiences include solving problems meeting needs and feeling warmth and care, while unpleasant service experiences include the formality that makes it difficult to benefit truly, lack of organization, regularity, sustainability, and the mismatch between service provision and actual demands. The needs for voluntary services mainly focuses on emotional comfort, Cultural and recreational, and knowledge acquisition.

Older adults in nursing home organizations have varied voluntary experiences, and their voluntary service needs are diversified. Voluntary service needs of older adults should be accurately assessed, and voluntary service activities should be focused upon.

Peer Review reports

Introduction

As a result of advancements in medical technology and improved sanitation conditions, the average life expectancy of Chinese people has increased significantly from 60 years in 1970 to 77.3 years in 2023. However, this has led to a growing number of older adults in China. According to the seventh population census conducted by the National Bureau of Statistics of China, there are now 260 million people over the age of 60 living in the country [ 1 ], The aging population in China is growing, and population balance is becoming a core challenge for the country in the long term. The increasing aging population has posed significant challenges and burdens to the state and society [ 2 ], China’s aging population challenges the current security system, requiring significant efforts from the state and society for improvement [ 3 ].

There are three main modes of old-age care in China: family old-age care, community old-age care, and institutionalized old-age care. Family old-age care is the most traditional form of old-age care in China, due to the reduction in family size and the formation of the “4-2-1” family model - which consists of four older adults, one couple, and one child - the traditional family model is no longer able to meet the growing demand for older adults care [ 4 ]; China’s community old-age care is still in the exploratory stage, facing challenges such as slow construction, insufficient staff, and lack of professional knowledge. As a result, it cannot provide meticulous care services for older adults [ 5 ]. Against this background, institutionalized older adult care has gradually become popular, it refers to older adults in social service organizations such as senior citizen apartments, welfare homes, and homes for older adults to spend their later life [ 6 ]. The challenges of population aging and the inadequacies of family and community support for older adults have resulted in a growing number of older adults opting to reside in nursing home organizations. This has undoubtedly placed additional burdens and challenges on these nursing home organizations. Due to multiple challenges such as late start, low quality, and lack of professional and technical talents, China’s nursing home organizations are still a long way from meeting the comprehensive needs of older adults in terms of health management, skilled nursing care, rehabilitation training, cultural and recreational services, psychological counseling, and social interaction [ 7 ]. To tackle the issue of an aging population in China and ensure that older adults have a high quality of life when choosing nursing home organizations, it is necessary to enhance the quality of older adult care services by engaging social forces, such as volunteer teams [ 8 ]. Voluntary services refer to the voluntary, unpaid public service offered by individuals, organizations, and voluntary service organizations to society or other organizations. The forms of voluntary services are diverse and can be either formal, planned, and long-term, or informal, spontaneous, and intermittent [ 9 ]. At present, volunteer groups in China’s nursing home organizations are mostly informal and consist of university students, healthcare workers, art workers, social workers, and others. These groups are invited by nursing home organizations or come to these institutions on their initiative to provide services for older adults. These services include a wide range of activities such as haircutting, cultural performances, spiritual comfort, hobby learning (e.g., paper-cutting, flower arranging), organizing festive activities (e.g., making rice dumplings on-site at Dragon Boat Festival, making mooncakes at Mid-Autumn Festival, etc.).

Voluntary services are a crucial aspect of long-term care and greatly complement the resources provided by the government,these nursing home organizations welcome volunteers who perform various non-medical activities associated with the daily lives of older adults [ 10 , 11 , 12 ]. Volunteers offer additional assistance and companionship to residents, provide support to employees such as nurses, nutritionists, and physical therapists, and potentially improve the overall quality of care, in China, these services have become increasingly popular and play a crucial role [ 13 , 14 ]. However, some problems have emerged in voluntary services, The voluntary services provided by volunteer organizations for older adults have certain functional defects and efficiency dilemmas, such as an unsound volunteer management system, high mobility of volunteers, and lack of a corresponding volunteer training system, which leads to the inability to provide high-quality services [ 13 ]. The above problems have undermined the effectiveness of voluntary services and affected the regular operation of nursing home organizations [ 15 ].

For effective services for older adults, it’s critical to understand the needs and experiences of older adults in nursing home organizations, there have been limited studies on how older adults feel about receiving voluntary services and if such services are suitable for their actual needs. One qualitative study documented the experiences of older adults who were helped by volunteers, but it was mainly focused on the volunteers themselves [ 16 ]. Another study looked into the benefits and experiences of receiving voluntary services, but it specifically focused on older adults who were confined to their homes [ 17 ]. There is no research available that sheds light on the emotions and requirements of older adults who receive voluntary services in nursing home organizations. To bridge this gap, we conducted interviews with older adults who have been accepted for voluntary services in two nursing home organizations in Hangzhou. The objective of this study is to gain a deeper understanding of the actual needs and experiences of older adults and use this information to guide promoting the effective growth of voluntary services and establishing a voluntary service system that is suitable for older adults in nursing home organizations.

This study adopts a qualitative descriptive approach to examine the experiences and expectations of older adults in nursing home organizations when receiving voluntary services. This study aims to gain a comprehensive understanding of the actual experiences and needs of older adults residing in nursing home organizations regarding receiving voluntary services and explore the types of voluntary services that are most suitable for the needs of older adults. To ensure accuracy and transparency, the authors followed the Consolidated Standards for Reporting Qualitative Research (COREQ) guidelines when reporting their findings [ 18 ].

Participants

During June-August 2023, the authors used purposive sampling to sample older adults residing in two nursing home organizations in Hangzhou, the inclusion criteria for the interview subjects were as follows:

they had to have resided in the nursing home organizations for more than a year;

they had to have received voluntary services;

they had to be conscious and able to express themselves effectively;

they had to have given informed consent and voluntarily agreed to participate in the study.

The number of people participating in the study was decided based on information saturation, this means the interviews were conducted until no new topics emerged and responses were repeated, the data from the twelfth interview indicated that saturation had been reached as confirmed by the other two interviews. This research principle was based on previous qualitative research studies [ 19 ]. A total of 14 older adults, coded N1-N14, were included in this study. All older adults who participated in the study agreed to the interview process, and none withdrew during the study. Detailed information can be found in Table  1 .

Interview outline

We developed an interview outline after thoroughly reviewing the literature sources and consulting with the research group [ 20 , 21 ]. We selected two older adults living in nursing home organizations to conduct pre-interviews, we adjusted the interview outline based on the feedback we received from the pre-interviews.

The interview will cover the following topics:

Please describe the voluntary services you have received in detail. How do you feel about receiving these services?

Are you satisfied with the voluntary service you have received? What aspects of the service make you satisfied?

What are your dissatisfactions with the voluntary service? Why do you feel that way?

What are your expectations and needs for the voluntary service’s content, form, and volunteers?

Is there anything else you would like to add to the discussion?

Data collection

A semi-structured interview method was utilized to gather data for this study. The main researcher, (a master’s degree nursing student) has been trained in qualitative research methods and has mastered the semi-structured interview techniques required to conduct interviews independently. Additionally, the researcher has participated in various volunteer activities in nursing organizations and has established a trustworthy relationship with the interviewees. Before conducting the interviews, the main researcher explained the study’s purpose and methodology to the interviewees and, after acquiring their consent, scheduled an appointment in advance. Face-to-face interviews were conducted with the respondents in a quiet, private, comfortable conference room. During the interview, the researcher recorded the entire process with the respondent’s consent without interrupting the respondent unnecessarily. The researcher confirmed the key concerns and the content that the respondent could not express clearly by repeating, asking follow-up questions, and asking rhetorical questions. The researcher also promptly recorded the respondent’s non-verbal information, such as movements, expressions, and tone of voice. Each interview lasted 30–45 min, and after conducting 14 interviews, no new information was obtained, indicating data saturation and ending the interview process. At the end of the interview, each interviewee was given a small token of appreciation.

Data analysis

The audio recordings of the interviews were transcribed into text within 24 h of completion, non-verbal information was noted in the transcript at relevant places. The transcribed information was then entered into the NVIVO 11.0 software (QST International, Cambridge, MA, USA) for data extraction, coding, and integration. Two researchers independently analyzed and coded the data, and the results were compared to identify common themes. Any discrepancies were resolved after the research team had discussed them to ensure that the data was complete and the analysis was accurate. Colaizzi’s seven-step analysis method was used to refine the themes from the interviews, which involved the following steps [ 22 ]:

Carefully read all the transcriptions of the interviews.

Analyze the significant statements made by the interviewees.

Code the recurring and meaningful ideas discussed in the interviews.

Gather the coded ideas and form the theme clusters.

Define and describe the themes from the coded ideas.

Identify similar ideas and sublimate the theme concepts.

Return the results to the interviewees for verification, and revision, and add the results based on the feedback from the interviewees. For detailed coding results, please see Table  2 .

After the data analysis was summarized, two main themes were identified: Experiences and Needs for volunteerism.

Theme 1: experiences

Beneficial experiences, solving problems and meeting needs.

Many older adults currently reside in nursing home organizations that are situated far away from their children and friends, they often face difficulties in getting help promptly when they encounter problems, which can affect their daily lives. For instance, in today’s rapidly developing society, many older adults own smartphones but lack the necessary knowledge to use them effectively. This, in turn, reduces their social participation and increases their sense of isolation. However, voluntary services have been instrumental in assisting them in overcoming these hurdles and leading a more fulfilling life.

N11: “When the volunteers come to teach me how to use computers, I ask them something that I don’t understand, and the teacher will explain it to me immediately.” N1: “I don’t know how to buy things online. Volunteers taught me little by little, and after a few teaching sessions, I learned how to do it so I don’t have to bother the caregiver every time. I can also do online shopping by myself, and I feel that life is much more convenient.”

Some respondents stated that volunteers could fulfill their needs. Professional volunteers also taught older adults Chinese medicine and health care and assisted with self-care.

N12: “I’m interested in Chinese medicine health care knowledge, and when students from the University of Traditional Chinese Medicine come over, and I ask them What are the functions of different acupoints, they tell me how to press them to make them work.”

Feel warmth and care

Many older adults live in nursing home organizations, away from their familiar environment and social network. This isolation can generate a sense of loneliness, making them more eager for emotional support. Volunteers provide services to add joy to the lives of older adults so that they feel cared for. Interviewees have mentioned that being taken care of on their initiative makes them feel warm and touched, increasing their overall sense of well-being.

N10: “I am delighted when I participate in volunteering, I feel that I have a group life again, I am pleased, I feel that someone cares about us.” N8: “Volunteers come to serve us, feel that people still care about us older adults, and now the country also cares about us, and society also cares about us, I am thrilled.”

Some respondents said that having someone to talk to and greet them would make them feel happy and that they were willing to communicate with young people and accept their new ideas.

N2: “As soon as I see you young people, I am happy, I feel the atmosphere of youth, my mood is different, I feel less lonely.”

Unpleasant service experiences

A formality that makes it difficult to benefit truly.

According to the interviewees, there are certain formalized phenomena in the domain of volunteering. Some volunteers engage in volunteering activities to obtain a certificate, such certificates can help them get extra points at work. Some volunteers participated in volunteering based on the mentality of the herd under the organizational arrangements of their schools or enterprises. These volunteers lack initiative, violate the principle of voluntarism, and cannot provide services that genuinely benefit older adults due to their single-mindedness and formalism during the service process. As a result, older adults have a poorer sense of experience.

N7: “Some volunteers are asked to serve by their companies, and they have to finish the job; some just go through a process.” N13: “Many volunteers come over to perform a show, then take photos and leave; the service time is very short, just like completing a task.” N5: “Some volunteers are very perfunctory; they come for a while and leave quickly.”

Lack of organization, regularity, sustainability

Many volunteers offer their services without compensation, while they have their formal jobs, which makes it difficult for them to provide services consistently. Additionally, volunteers may be more mobile, which can result in a lack of continuity in the services that are provided and the target groups that are served. However, older adults living in nursing home organizations often have monotonous and lonely lives, and occasional voluntary services may not be enough to meet their needs. As a result, some older adults may feel dissatisfied with the irregular and unsustainable nature of voluntary services.

N12: “Volunteers come on an ad hoc basis; they are not regular. Recently, a school teacher came to teach us how to sing, but unfortunately, they had to leave due to commitments and have not been able to come back.’’ N5: “Volunteers can’t come regularly; they come once in a while or not regularly and don’t have a plan.” N7: “Volunteers come to the nursing home occasionally, so they don’t want to bother them.”

The mismatch between service provision and actual demand

The voluntary services provided to older adults in nursing institutions were not able to match their real needs as the volunteers had no prior knowledge of their needs and did not make any advance preparations.

N4: “Last time, a volunteer came and asked me if I needed help with cleaning. However, I declined their kind offer because caregivers in the nursing home clean rooms every day, and the volunteers could not address the specific things I needed help with.”

The needs of older adults for volunteering can vary significantly based on their experiential backgrounds, and physiological and psychological conditions. Therefore, providing the same services to all older adults can lead to negative feelings towards volunteering among them.

N10: “Some volunteers come just to dance and sing, it feels very noisy. I don’t want to participate, I want the volunteers to talk to me peacefully and quietly.” N14: “I am not very good with my legs, so it is difficult for me to participate in activities organized by the volunteers downstairs. I would like to find activities I can participate in in my room, such as playing games or doing crafts.”

Theme 2 needs for volunteerism

Needs for emotional comfort.

Many older adults live in semi-closed institutions where they lack long-term support from their families and struggle to find someone to talk to. During the epidemic, nursing home organizations prohibited visitors to prevent the spread of the virus, leaving many seniors alone and cut off from the outside world. As a result, many older adults experience feelings of loneliness and depression. To help combat these negative emotions, volunteers can provide companionship and support, which can effectively reduce feelings of loneliness and promote emotional well-being.

N1: “I hope someone will come and chat with us; many older adults have no way to contact the outside world, so they have psychological barriers, they need psychological counseling, they need someone to come and chat with them to relieve their loneliness.” N10: “It’s better to have volunteers to come over to the service, to come and chat with me, to visit me.” N12: “I would like volunteers to communicate with us, tell us what is happening outside, tell us something new.”

Cultural and recreational needs

As people age, their social interactions tend to decrease, and they gradually tend to withdraw from daily life. This results in older adults having more free time after their retirement. Nursing home organizations can provide basic living care and medical assistance for older adults, which relieves them of the burden of cooking, cleaning, and shopping. This also means they have more free time than those who live at home or in the community. Many older adults wish to participate in cultural and recreational activities, such as singing, dancing, sports, and watching performances, to add excitement to their lives. They hope that volunteers can organize such activities to help them reduce their loneliness and spend their time in a meaningful way.

N14: “It’s good for volunteers to come and teach us how to dance, sing, and sing opera, and time passes a little faster when we all get together and learn.” N2: “It is popular for volunteers to bring cultural performances to our nursing home, we love to see young people performing programs, singing some classic old songs or Peking Opera, it is very popular.” N9: “We would like to play tai chi, it is a very suitable sport for us as it strengthens the body and the movements are softer, it would be nice if a teacher could teach us.”

Knowledge acquisition needs

According to Maslow’s Hierarchy of Needs Theory, individuals will naturally shift their focus toward higher-level pursuits once their basic and low-level needs are met. In the case of older adults residing in nursing home organizations, their basic material needs are taken care of, and as a result, their need for knowledge and learning becomes increasingly important. Many older adults require assistance in learning how to use electronic equipment, which can help facilitate their communication with the outside world and reduce feelings of isolation.

N1: “It’s become very convenient to buy things online, but I don’t know how to operate it myself and would like someone to teach me.” N2: “My daughter bought me an expensive Apple phone, but I am unfamiliar with how to use it. It would be great if someone could systematically instruct me on how to use the smartphone.” N8: “I don’t know how to use my smartphone, I don’t understand many functions, so I would benefit from having a teacher to guide me.”

As individuals age, their bodily and cognitive functions may deteriorate, adversely affecting their quality of life. Basic healthcare knowledge can be critical for older adults to maintain good health. Many older adults have a strong desire to learn about nutritional diets, rational exercise, and traditional Chinese medicine physiotherapy as a means of improving their health.

N9: “Volunteers can come and talk to us about medicine and how to predict dementia.” N13: “I have high blood pressure and cholesterol. I need advice on what to eat and what to avoid.”

To prevent any disagreements regarding the distribution of their assets among their heirs after they pass away, older adults seek the help of volunteers to assist them in drafting a will that is by national policies and regulations and has legal validity.

N12: “Volunteers can help us learn how to write a will effectively and can avoid unnecessary trouble and conflicts in the future.”

The current situation of voluntary experiences of older adults in nursing home organizations

Analysis of beneficial experiences.

The study’s findings indicate that individuals residing in nursing home organizations who are of advanced age have mixed experiences when it comes to receiving voluntary services. Most respondents conveyed the warmth and care emanating from the volunteers and the society towards older adults. Furthermore, they shared that volunteering offered them a means to engage in activities actively, create connections with fellow older adults, and foster mutual support and camaraderie. This social participation has the potential to enhance the mental well-being of older adults, thereby decreasing feelings of loneliness and depression [ 17 ]. Voluntary activities like smartphone training can help older adults acquire the necessary needed skills and adapt better to modern technology and life. Competent skills are crucial for older adults, particularly in today’s fast-developing technological society, where electronic devices such as smartphones are becoming increasingly popular. However, many older adults need more skills to operate these devices and thus cannot fully utilize them. Through training, older adults can learn how to use smartphones, including sending text messages, browsing the web, using social media, downloading applications, and more. Learning these skills not only improves the quality of life of older adults but also helps them stay connected with family and friends, thereby reducing loneliness.

Improved skills can assist older adults in accessing and utilizing health information, including online medical advice and health apps. This information can aid in managing their health status, preventing and managing chronic illnesses, and ultimately improving their quality of life. Volunteering is crucial in nursing home organizations. It provides numerous benefits to older adults, including enhancing their mental health and quality of life and receiving the necessary support and care by participating in voluntary activities [ 23 ].

Analysis of unpleasant experiences

During the interviews, some older adults shared negative experiences regarding the content, form, and frequency of voluntary services. They pointed out that volunteers did not understand their needs in advance, focusing too much on material assistance and neglecting their psychological and intellectual needs. Additionally, the service process is often too process-oriented and formalized, with less interaction with older adults, resulting in voluntary services failing to meet their expectations.

Research suggests that negative experiences of receiving voluntary services may impact older adults’ willingness to seek help and the effectiveness of voluntary services. Therefore, when providing voluntary services to older adults, it is essential to take the initiative to understand their experiences and continuously optimize the voluntary program. This approach is crucial to improving the quality of voluntary services [ 24 ].

The current situation of the demands for voluntary services by older adults

The study results show that nursing home organizations can provide comprehensive life care services to older adults, meaning they do not require many voluntary services for life care. However, this does not imply that older adults’ needs are met. Their need for emotional support, cultural recreation, and knowledge-seeking and learning is highly concentrated.

When older adults leave their familiar family environment to move into care institutions, they may experience feelings of loneliness and boredom due to the lack of regular interaction with their children, family members, and friends. This sense of isolation can harm their mental health, and they may seek more opportunities to communicate and interact with younger individuals to gain emotional comfort [ 25 ].

As people age, cultural entertainment and knowledge learning become essential for spiritual growth. After their basic living needs are taken care of, older adults desire more fulfilling recreational activities, such as calligraphy, painting, and singing, these activities enrich the spiritual life of older adults and benefit their physical and mental health [ 26 ].

In today’s rapidly developing society, the widespread use of smartphones and the popularity of online shopping have led to a digital divide among older adults. This phenomenon has, to some extent, hindered their social participation and increased their sense of isolation. Consequently, there is a growing demand for voluntary services that assist with smartphone use and can help them enjoy a convenient and fulfilling digital life.

The need for voluntary services for older adults has changed over time. While they still require help with their daily living, they also need emotional support, cultural engagement, and opportunities to learn new things. We should focus on meeting these needs to ensure our voluntary services are beneficial. By doing so, we can help older adults live fulfilling, healthy, and happy lives in their later years [ 27 ].

Suggestions and strategies for optimizing volunteerism

Accurately assessing older adults’ voluntary service needs.

The study results reveal that some older adults have negative experiences with voluntary services that fail to meet their actual needs, leading to unsatisfactory service outcomes. This highlights the need to accurately identify the real service needs of older adults to improve the quality and effectiveness of voluntary services.

To achieve our goal, we need to take a series of steps. Firstly, we must create appropriate tools for evaluating the needs of older adults for voluntary services. We should also clarify the assessment methods and strategies for assessing these needs, before launching voluntary services, relevant organizations and volunteers must understand older adults’ service experience and needs through qualitative and quantitative assessment methods [ 28 ].

To improve the quality and effectiveness of voluntary services for older adults, we can utilize big data technology to carry out precise reforms. This involves building a unified information platform for voluntary services that enables a quick match between the needs of older adults and the specialties of volunteers through the co-construction, sharing, and everyday use of resource information [ 29 ]. By doing so, we can provide multi-level, multi-category, and personalized voluntary services that cater to the actual needs of older adults, thus achieving the purpose of “precise service.”

In conclusion, we must prioritize the actual needs of older adults and provide them with more personalized and intimate voluntary services by continuously improving the assessment tools and information platforms with the orientation of precise services, the use of big data technology will play a key role in helping us realize the goal of efficient and accurate services.

Improving the quality management system of voluntary services

Volunteering quality refers to the quality of services volunteers provide, as perceived by the direct recipients. Research has shown that low-quality voluntary services fail to achieve their intended goals, moreover, negative experiences of receiving voluntary services may discourage older adults from seeking help in the future. The study highlights a significant gap between older adults’ experience of volunteering quality and their expectations, therefore, it is necessary to strengthen the management of volunteering quality to ensure that expectations are met.

To enhance the quality of volunteering, we need to implement measures. Firstly, we must optimize the recruitment and selection system for volunteers, this entails formulating recruitment plans and selection requirements that align with the voluntary services needs of older adults. We aim to create a stable and committed volunteer team skilled in services knowledge and job skills and willing to participate in voluntary services for an extended period [ 30 ].

To enhance the level and quality of service, it is important to provide regular and standardized training to volunteers. Volunteers should receive professional information support services, such as training on volunteer spirit, etiquette, communication skills, and the physiological and psychological characteristics of older adults. The main forms of training include information consultation, professional knowledge, technology lectures, sharing of previous volunteer experiences, summarizing stage-by-stage voluntary services, and experiential services. Volunteers should be provided with face-to-face or online interaction to help them improve their ability to assist older adults. The training for volunteering encompasses theoretical knowledge about volunteering, including its characteristics and principles, the rights and interests of service users, and respect for them. It also includes basic knowledge of social work, such as interpersonal communication methods and skills, as well as knowledge of health care for older adults. The latter includes the introduction of general knowledge about daily life care for older adults, such as diet, hygiene, and exercise, and the evaluation of the training’s effectiveness. Both voluntary service organizations and nursing home organizations should participate in the training process, only volunteers who have completed the training and assessment can engage in service activities [ 31 ]. It is essential to improve the evaluation mechanism of voluntary service quality. This can be done by creating a scientific evaluation index system involving older adults in evaluating their satisfaction with the voluntary service program and conducting a comprehensive analysis of the evaluation results. This analysis can help to optimize and improve the service program, additionally, tracking and evaluating the effectiveness of optimization measures to continuously enhance service quality is crucial [ 32 ].

Improving the quality of voluntary services is a comprehensive project that enhances various aspects, such as volunteer recruitment, training, and service quality evaluation. This systematic approach can help serve the nursing home organizations better and improve their overall quality of life.

Strengths and limitations

The paper’s strength lies in its focus on the experience of older adults in nursing institutions when receiving voluntary services and their need for such services. This study’s understanding of the real feelings and needs of older adults is beneficial for various organizations in society to provide better services in a targeted manner. However, the study’s limitation is that it mainly focuses on the more developed areas of Hangzhou, which affects the sample’s representativeness and makes it challenging to reflect the general situation of older adults in nursing home organizations. Additionally, the author’s subjective viewpoints may affect the analysis of the material during the data analysis process. Finally, the sample size of this study is relatively small, and there may be individual differences in personality, physical condition, and economic situation, among others. Therefore, expanding the sample size and the region’s scope to carry out more in-depth research is necessary.

This research explored the experiences and requirements of older adults who receive voluntary services in Chinese care homes. The study categorized their experiences into two groups: beneficial experiences and unpleasant service experiences, the needs of older adults who receive voluntary services include emotional comfort, cultural and recreational, and knowledge acquisition. It is crucial to have a timely and comprehensive understanding of the experiences and needs of older adults to create a targeted voluntary service model, standardized management, and training of volunteers in nursing home organizations.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request. The datasets are not publicly available due to confidentiality and ethical restrictions.

Zhou Y, LI Y, Zhu X, et al. Medical and old-age care integration model and implementation of the Integrated Care of Older people (ICOPE) in China: opportunities and challenges [J]. J Nutr Health Aging. 2021;25(6):720–3. https://doi.org/10.1007/s12603-021-1595-5 .

Article   CAS   PubMed   Google Scholar  

National Bureau of Statistics. Bulletin of the Seventh National Census. Published online May 11. 2021. Accessed October 22, 2023. http://www.stats.gov.cn/ztjc/zdtjgz/zgrkpc/dqcrkpc/index.html . http://www.stats.gov.cn/ztjc/zdtjgz/zgrkpc/dqcrkpc/index.html.

Duan L, Liu Z, Yu W, et al. The provincial trend of population aging in China—based on population expansion forecast formula. J Comput Methods Sci Eng. 2022;22(1):349–59. https://doi.org/10.3233/jcm-215630 .

Article   Google Scholar  

Sun J, Guo Y, Wang X, Zeng Q. mHealth for aging China: opportunities and challenges. Aging Dis. 2016;7(1):53–67. https://doi.org/10.14336/AD.2015.1011 .

Article   PubMed   PubMed Central   Google Scholar  

Feng Z, Liu C, Guan X, Mor V. China’s rapidly aging population creates policy challenges in shaping a viable long-term care system. Health Aff (Millwood). 2012;31(12):2764–73. https://doi.org/10.1377/hlthaff.2012.0535 .

Article   PubMed   Google Scholar  

Hu H, Si Y, Li B. Decomposing inequality in long-term care need among older adults with chronic diseases in China: a life course perspective. Int J Environ Res Public Health. 2020;17(7):2559. https://doi.org/10.3390/ijerph17072559 .

China Volunteer Service Website. Basic norms of volunteer service organizations. Published online June 29. 2021. https://chinavolunteer.mca.gov.cn/NVSI/LEAP/site/index.html%23/newsinfo/1/880854ca8766433183498e7bf819e879 .

Hansen T, Slagsvold B. An Army of Volunteers? Engagement, motivation, and barriers to volunteering among the baby boomers. J Gerontol Soc Work. 2020;63(4):335–53. https://doi.org/10.1080/01634372.2020.1758269 .

Crittenden JA, Coleman RL, Butler SS. It helps me find balance: older adult perspectives on the intersection of caregiving and volunteering. Home Health Care Serv Q. 2022;1–19. https://doi.org/10.1080/01621424.2022.2034700 . Published online January 30, 2022.

Pickell Z, Gu K, Williams AM. Virtual volunteers: the importance of restructuring medical volunteering during the COVID-19 pandemic. Med Humanit. 2020; 46:537?40. https://doi.org/10.1136/medhum-2020-011956 .

Ayton D, O’Donnell R, Vicary D, et al. Psychosocial volunteer support for older adults with cognitive impairment: development of MyCare ageing using a codesign approach via action research. BMJ Open. 2020;10:e036449. https://doi.org/10.1136/bmjopen-2019-036449 .

Fearn M, Harper R, Major G, et al. Befriending older adults in nursing homes: volunteer perceptions of switching to remote befriending in the COVID-19 era. Clin Gerontol. 2021;44:430–8. https://doi.org/10.1080/07317115.2020.1868646 .

Yang M, Liu K, Lu L, et al. Investigation on the status quo and needs of pension institutions receiving volunteer service of college students. J Nurs Res. 2019;33(23):4156–60.

Google Scholar  

Zhang QZ, Yang LL. Status quo investigation of geriatric nursing volunteer service in a university in Kaohsiung, Taiwan. J Nurs Res. 2017;31(36):4707–9.

Liu Y, Duan Y, Xu L. Volunteer service and positive attitudes toward aging among Chinese older adults: the mediating role of health. Soc Sci Med. 2020;265:113535. https://doi.org/10.1016/j.socscimed.2020.113535 .

Ryninks K, Wallace V, Gregory JD. Older adult hoarders’ experiences of being helped by volunteers and volunteers’ experiences of helping. Behav Cogn Psychother. 2019;47(6):697–708. https://doi.org/10.1017/S135246581900016X .

You YJ, Hang L, Liu YL, et al. Qualitative study on lived experience and needs of humanistic care among the housebound elderly with special difficulties in Wuhan [J]. J Nurs Sci. 2022;37(04):85–8.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57. https://doi.org/10.1093/intqhc/mzm042 .

Saunders B, Sim J, Kingstone T, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52(4):1893–907. https://doi.org/10.1007/s11135-017-0574-8 .

Huang L, Zhang F, Guo L, et al. Experiences and expectations of receiving volunteer services among home-based elderly in Chinese urban areas: a qualitative study [J]. Health Expect. 2022;25(6):3164–74. https://doi.org/10.1111/hex.13624 .

Liu M. The application of Colaizzi’s seven steps in the analysis of phenomenological research data [J]. J Nurs Sci. 2019;34(11):90–2.

Matthews K, Nazroo J. The impact of Volunteering and its characteristics on Well-being after State Pension Age: longitudinal evidence from the English Longitudinal Study of ageing [J]. J Gerontol B Psychol Sci Soc Sci. 2021;76(3):632–41. https://doi.org/10.1093/geronb/gbaa146 .

Newbigging K, Mohan J, Rees J, et al. Contribution of the voluntary sector to mental health crisis care in England: protocol for a multimethod study [J]. BMJ Open. 2017;7(11):e019238. https://doi.org/10.1136/bmjopen-2017-019238 .

Kunin M, Advocat J, Gunatillaka N, Russell G. Access to care: a qualitative study exploring the primary care needs and experiences of older people needing assistance with daily living. Aust J Prim Health. 2021;27(3):228–35. https://doi.org/10.1071/PY20180 .

Gardiner C, Laud P, Heaton T, et al. What is the prevalence of loneliness amongst older people living in residential and nursing care homes? A systematic review and meta-analysis [J]. Age Ageing. 2020;49(5):748–57. https://doi.org/10.1093/ageing/afaa049 .

Lapane KL, Lim E, Mcphillips E, et al. Health effects of loneliness and social isolation in older adults living in congregate long-term care settings: a systematic review of quantitative and qualitative evidence [J]. Arch Gerontol Geriatr. 2022;102:104728. https://doi.org/10.1016/j.archger.2022.104728 .

Plattner L, Brandstötter C, Paal P. [Loneliness in nursing homes-experience and measures for amelioration: a literature review] [J]. Z Gerontol Geriatr. 2022;55(1):5–10. https://doi.org/10.1007/s00391-021-01881-z .

Zhang M, Chen C, Du Y, Wang S, Rask M. Multidimensional factors affecting care needs in daily living among community-dwelling older adults: a structural equation modelling approach. J Nurs Manag. https://doi.org/10.1111/jo nm.13259.

Gong N, Meng Y, Hu Q, et al. Obstacles to access to community care in urban senior-only households: a qualitative study. BMC Geriatr. 2022;22(1):122. https://doi.org/10.1186/s12877-022-02816-y .

Millette V, Gagné M. Designing volunteers’ tasks to maximize motivation, satisfaction, and performance: the impact of job characteristics on volunteer engagement. Motiv Emot. 2008;32(1):11–22.

Siqueira MAM, Torsani MB, Gameiro GR, et al. Medical students’ participation in the volunteering program during the COVID-19 pandemic: a qualitative study about motivation and the development of new competencies. BMC Med Educ. 2022;22(1):111. https://doi.org/10.1186/s12909-022-03147-7 .

Bussell H, Forbes D. Developing relationship marketing in the voluntary sector. J Nonprofit Public Sect Mark. 2006;7(3):244–57.

Download references

Acknowledgements

We want to express our heartfelt appreciation to the 14 older adults who participated in the interview and shared their experiences. We are also grateful to the administrators of nursing home organizations in Hangzhou, Zhejiang Province, for granting us access and allowing us to conduct the interviews at their facility. Their cooperation was invaluable in gaining insights into the needs of older adults.

This study did not receive any form of financial support.

Author information

Authors and affiliations.

School of Nursing, Zhejiang Chinese Medical University, 548 Binwen Road, Binjiang District, Hangzhou, 310053, Zhejiang, China

Lin Li, Qin Shen & Junxian Wu

You can also search for this author in PubMed   Google Scholar

Contributions

Li and Wu were responsible for data collection, sorting, and analysis, and Li wrote the paper. Shen directed and revised the article and approved the final version for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Qin Shen .

Ethics declarations

Ethics approval and consent to participate.

The study protocol was approved by the Medical Ethics Committee of Zhejiang Chinese Medical University (approval No. 20230814-2). Before the interviews, the participants were provided with information regarding the study’s purpose and procedures, the voluntary nature of their participation, and the confidentiality of their data. The interview data was stored securely, and only the research team could access it. The Ethics Committee of Zhejiang Chinese Medical University approved this study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Li, L., Shen, Q. & Wu, J. The experiences and needs of older adults receiving voluntary services in Chinese nursing home organizations: a qualitative study. BMC Health Serv Res 24 , 547 (2024). https://doi.org/10.1186/s12913-024-11045-5

Download citation

Received : 25 December 2023

Accepted : 25 April 2024

Published : 29 April 2024

DOI : https://doi.org/10.1186/s12913-024-11045-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Nursing home organizations
• Qualitative research

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]