case study for research methodology

• Privacy Policy

Home » Case Study – Methods, Examples and Guide

Case Study – Methods, Examples and Guide

Table of Contents

A case study is a research method that involves an in-depth examination and analysis of a particular phenomenon or case, such as an individual, organization, community, event, or situation.

It is a qualitative research approach that aims to provide a detailed and comprehensive understanding of the case being studied. Case studies typically involve multiple sources of data, including interviews, observations, documents, and artifacts, which are analyzed using various techniques, such as content analysis, thematic analysis, and grounded theory. The findings of a case study are often used to develop theories, inform policy or practice, or generate new research questions.

Types of Case Study

Types and Methods of Case Study are as follows:

Single-Case Study

A single-case study is an in-depth analysis of a single case. This type of case study is useful when the researcher wants to understand a specific phenomenon in detail.

For Example , A researcher might conduct a single-case study on a particular individual to understand their experiences with a particular health condition or a specific organization to explore their management practices. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of a single-case study are often used to generate new research questions, develop theories, or inform policy or practice.

Multiple-Case Study

A multiple-case study involves the analysis of several cases that are similar in nature. This type of case study is useful when the researcher wants to identify similarities and differences between the cases.

For Example, a researcher might conduct a multiple-case study on several companies to explore the factors that contribute to their success or failure. The researcher collects data from each case, compares and contrasts the findings, and uses various techniques to analyze the data, such as comparative analysis or pattern-matching. The findings of a multiple-case study can be used to develop theories, inform policy or practice, or generate new research questions.

Exploratory Case Study

An exploratory case study is used to explore a new or understudied phenomenon. This type of case study is useful when the researcher wants to generate hypotheses or theories about the phenomenon.

For Example, a researcher might conduct an exploratory case study on a new technology to understand its potential impact on society. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as grounded theory or content analysis. The findings of an exploratory case study can be used to generate new research questions, develop theories, or inform policy or practice.

Descriptive Case Study

A descriptive case study is used to describe a particular phenomenon in detail. This type of case study is useful when the researcher wants to provide a comprehensive account of the phenomenon.

For Example, a researcher might conduct a descriptive case study on a particular community to understand its social and economic characteristics. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of a descriptive case study can be used to inform policy or practice or generate new research questions.

Instrumental Case Study

An instrumental case study is used to understand a particular phenomenon that is instrumental in achieving a particular goal. This type of case study is useful when the researcher wants to understand the role of the phenomenon in achieving the goal.

For Example, a researcher might conduct an instrumental case study on a particular policy to understand its impact on achieving a particular goal, such as reducing poverty. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of an instrumental case study can be used to inform policy or practice or generate new research questions.

Case Study Data Collection Methods

Here are some common data collection methods for case studies:

Interviews involve asking questions to individuals who have knowledge or experience relevant to the case study. Interviews can be structured (where the same questions are asked to all participants) or unstructured (where the interviewer follows up on the responses with further questions). Interviews can be conducted in person, over the phone, or through video conferencing.

Observations

Observations involve watching and recording the behavior and activities of individuals or groups relevant to the case study. Observations can be participant (where the researcher actively participates in the activities) or non-participant (where the researcher observes from a distance). Observations can be recorded using notes, audio or video recordings, or photographs.

Documents can be used as a source of information for case studies. Documents can include reports, memos, emails, letters, and other written materials related to the case study. Documents can be collected from the case study participants or from public sources.

Surveys involve asking a set of questions to a sample of individuals relevant to the case study. Surveys can be administered in person, over the phone, through mail or email, or online. Surveys can be used to gather information on attitudes, opinions, or behaviors related to the case study.

Artifacts are physical objects relevant to the case study. Artifacts can include tools, equipment, products, or other objects that provide insights into the case study phenomenon.

How to conduct Case Study Research

Conducting a case study research involves several steps that need to be followed to ensure the quality and rigor of the study. Here are the steps to conduct case study research:

• Define the research questions: The first step in conducting a case study research is to define the research questions. The research questions should be specific, measurable, and relevant to the case study phenomenon under investigation.
• Select the case: The next step is to select the case or cases to be studied. The case should be relevant to the research questions and should provide rich and diverse data that can be used to answer the research questions.
• Collect data: Data can be collected using various methods, such as interviews, observations, documents, surveys, and artifacts. The data collection method should be selected based on the research questions and the nature of the case study phenomenon.
• Analyze the data: The data collected from the case study should be analyzed using various techniques, such as content analysis, thematic analysis, or grounded theory. The analysis should be guided by the research questions and should aim to provide insights and conclusions relevant to the research questions.
• Draw conclusions: The conclusions drawn from the case study should be based on the data analysis and should be relevant to the research questions. The conclusions should be supported by evidence and should be clearly stated.
• Validate the findings: The findings of the case study should be validated by reviewing the data and the analysis with participants or other experts in the field. This helps to ensure the validity and reliability of the findings.
• Write the report: The final step is to write the report of the case study research. The report should provide a clear description of the case study phenomenon, the research questions, the data collection methods, the data analysis, the findings, and the conclusions. The report should be written in a clear and concise manner and should follow the guidelines for academic writing.

Examples of Case Study

Here are some examples of case study research:

• The Hawthorne Studies : Conducted between 1924 and 1932, the Hawthorne Studies were a series of case studies conducted by Elton Mayo and his colleagues to examine the impact of work environment on employee productivity. The studies were conducted at the Hawthorne Works plant of the Western Electric Company in Chicago and included interviews, observations, and experiments.
• The Stanford Prison Experiment: Conducted in 1971, the Stanford Prison Experiment was a case study conducted by Philip Zimbardo to examine the psychological effects of power and authority. The study involved simulating a prison environment and assigning participants to the role of guards or prisoners. The study was controversial due to the ethical issues it raised.
• The Challenger Disaster: The Challenger Disaster was a case study conducted to examine the causes of the Space Shuttle Challenger explosion in 1986. The study included interviews, observations, and analysis of data to identify the technical, organizational, and cultural factors that contributed to the disaster.
• The Enron Scandal: The Enron Scandal was a case study conducted to examine the causes of the Enron Corporation’s bankruptcy in 2001. The study included interviews, analysis of financial data, and review of documents to identify the accounting practices, corporate culture, and ethical issues that led to the company’s downfall.
• The Fukushima Nuclear Disaster : The Fukushima Nuclear Disaster was a case study conducted to examine the causes of the nuclear accident that occurred at the Fukushima Daiichi Nuclear Power Plant in Japan in 2011. The study included interviews, analysis of data, and review of documents to identify the technical, organizational, and cultural factors that contributed to the disaster.

Application of Case Study

Case studies have a wide range of applications across various fields and industries. Here are some examples:

Business and Management

Case studies are widely used in business and management to examine real-life situations and develop problem-solving skills. Case studies can help students and professionals to develop a deep understanding of business concepts, theories, and best practices.

Case studies are used in healthcare to examine patient care, treatment options, and outcomes. Case studies can help healthcare professionals to develop critical thinking skills, diagnose complex medical conditions, and develop effective treatment plans.

Case studies are used in education to examine teaching and learning practices. Case studies can help educators to develop effective teaching strategies, evaluate student progress, and identify areas for improvement.

Social Sciences

Case studies are widely used in social sciences to examine human behavior, social phenomena, and cultural practices. Case studies can help researchers to develop theories, test hypotheses, and gain insights into complex social issues.

Law and Ethics

Case studies are used in law and ethics to examine legal and ethical dilemmas. Case studies can help lawyers, policymakers, and ethical professionals to develop critical thinking skills, analyze complex cases, and make informed decisions.

Purpose of Case Study

The purpose of a case study is to provide a detailed analysis of a specific phenomenon, issue, or problem in its real-life context. A case study is a qualitative research method that involves the in-depth exploration and analysis of a particular case, which can be an individual, group, organization, event, or community.

The primary purpose of a case study is to generate a comprehensive and nuanced understanding of the case, including its history, context, and dynamics. Case studies can help researchers to identify and examine the underlying factors, processes, and mechanisms that contribute to the case and its outcomes. This can help to develop a more accurate and detailed understanding of the case, which can inform future research, practice, or policy.

Case studies can also serve other purposes, including:

• Illustrating a theory or concept: Case studies can be used to illustrate and explain theoretical concepts and frameworks, providing concrete examples of how they can be applied in real-life situations.
• Developing hypotheses: Case studies can help to generate hypotheses about the causal relationships between different factors and outcomes, which can be tested through further research.
• Providing insight into complex issues: Case studies can provide insights into complex and multifaceted issues, which may be difficult to understand through other research methods.
• Informing practice or policy: Case studies can be used to inform practice or policy by identifying best practices, lessons learned, or areas for improvement.

Advantages of Case Study Research

There are several advantages of case study research, including:

• In-depth exploration: Case study research allows for a detailed exploration and analysis of a specific phenomenon, issue, or problem in its real-life context. This can provide a comprehensive understanding of the case and its dynamics, which may not be possible through other research methods.
• Rich data: Case study research can generate rich and detailed data, including qualitative data such as interviews, observations, and documents. This can provide a nuanced understanding of the case and its complexity.
• Holistic perspective: Case study research allows for a holistic perspective of the case, taking into account the various factors, processes, and mechanisms that contribute to the case and its outcomes. This can help to develop a more accurate and comprehensive understanding of the case.
• Theory development: Case study research can help to develop and refine theories and concepts by providing empirical evidence and concrete examples of how they can be applied in real-life situations.
• Practical application: Case study research can inform practice or policy by identifying best practices, lessons learned, or areas for improvement.
• Contextualization: Case study research takes into account the specific context in which the case is situated, which can help to understand how the case is influenced by the social, cultural, and historical factors of its environment.

Limitations of Case Study Research

There are several limitations of case study research, including:

• Limited generalizability : Case studies are typically focused on a single case or a small number of cases, which limits the generalizability of the findings. The unique characteristics of the case may not be applicable to other contexts or populations, which may limit the external validity of the research.
• Biased sampling: Case studies may rely on purposive or convenience sampling, which can introduce bias into the sample selection process. This may limit the representativeness of the sample and the generalizability of the findings.
• Subjectivity: Case studies rely on the interpretation of the researcher, which can introduce subjectivity into the analysis. The researcher’s own biases, assumptions, and perspectives may influence the findings, which may limit the objectivity of the research.
• Limited control: Case studies are typically conducted in naturalistic settings, which limits the control that the researcher has over the environment and the variables being studied. This may limit the ability to establish causal relationships between variables.
• Time-consuming: Case studies can be time-consuming to conduct, as they typically involve a detailed exploration and analysis of a specific case. This may limit the feasibility of conducting multiple case studies or conducting case studies in a timely manner.
• Resource-intensive: Case studies may require significant resources, including time, funding, and expertise. This may limit the ability of researchers to conduct case studies in resource-constrained settings.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Questionnaire – Definition, Types, and Examples

Observational Research – Methods and Guide

Quantitative Research – Methods, Types and...

Qualitative Research Methods

Explanatory Research – Types, Methods, Guide

Survey Research – Types, Methods, Examples

The Ultimate Guide to Qualitative Research - Part 1: The Basics

• Introduction and overview
• What is qualitative research?
• What is qualitative data?
• Examples of qualitative data
• Qualitative vs. quantitative research
• Mixed methods
• Qualitative research preparation
• Theoretical perspective
• Theoretical framework
• Literature reviews

Research question

• Conceptual framework
• Conceptual vs. theoretical framework

Data collection

• Qualitative research methods
• Focus groups
• Observational research

What is a case study?

Applications for case study research, what is a good case study, process of case study design, benefits and limitations of case studies.

• Ethnographical research
• Ethical considerations
• Confidentiality and privacy
• Power dynamics
• Reflexivity

Case studies

Case studies are essential to qualitative research , offering a lens through which researchers can investigate complex phenomena within their real-life contexts. This chapter explores the concept, purpose, applications, examples, and types of case studies and provides guidance on how to conduct case study research effectively.

Whereas quantitative methods look at phenomena at scale, case study research looks at a concept or phenomenon in considerable detail. While analyzing a single case can help understand one perspective regarding the object of research inquiry, analyzing multiple cases can help obtain a more holistic sense of the topic or issue. Let's provide a basic definition of a case study, then explore its characteristics and role in the qualitative research process.

Definition of a case study

A case study in qualitative research is a strategy of inquiry that involves an in-depth investigation of a phenomenon within its real-world context. It provides researchers with the opportunity to acquire an in-depth understanding of intricate details that might not be as apparent or accessible through other methods of research. The specific case or cases being studied can be a single person, group, or organization – demarcating what constitutes a relevant case worth studying depends on the researcher and their research question .

Among qualitative research methods , a case study relies on multiple sources of evidence, such as documents, artifacts, interviews , or observations , to present a complete and nuanced understanding of the phenomenon under investigation. The objective is to illuminate the readers' understanding of the phenomenon beyond its abstract statistical or theoretical explanations.

Characteristics of case studies

Case studies typically possess a number of distinct characteristics that set them apart from other research methods. These characteristics include a focus on holistic description and explanation, flexibility in the design and data collection methods, reliance on multiple sources of evidence, and emphasis on the context in which the phenomenon occurs.

Furthermore, case studies can often involve a longitudinal examination of the case, meaning they study the case over a period of time. These characteristics allow case studies to yield comprehensive, in-depth, and richly contextualized insights about the phenomenon of interest.

The role of case studies in research

Case studies hold a unique position in the broader landscape of research methods aimed at theory development. They are instrumental when the primary research interest is to gain an intensive, detailed understanding of a phenomenon in its real-life context.

In addition, case studies can serve different purposes within research - they can be used for exploratory, descriptive, or explanatory purposes, depending on the research question and objectives. This flexibility and depth make case studies a valuable tool in the toolkit of qualitative researchers.

Remember, a well-conducted case study can offer a rich, insightful contribution to both academic and practical knowledge through theory development or theory verification, thus enhancing our understanding of complex phenomena in their real-world contexts.

What is the purpose of a case study?

Case study research aims for a more comprehensive understanding of phenomena, requiring various research methods to gather information for qualitative analysis . Ultimately, a case study can allow the researcher to gain insight into a particular object of inquiry and develop a theoretical framework relevant to the research inquiry.

Why use case studies in qualitative research?

Using case studies as a research strategy depends mainly on the nature of the research question and the researcher's access to the data.

Conducting case study research provides a level of detail and contextual richness that other research methods might not offer. They are beneficial when there's a need to understand complex social phenomena within their natural contexts.

The explanatory, exploratory, and descriptive roles of case studies

Case studies can take on various roles depending on the research objectives. They can be exploratory when the research aims to discover new phenomena or define new research questions; they are descriptive when the objective is to depict a phenomenon within its context in a detailed manner; and they can be explanatory if the goal is to understand specific relationships within the studied context. Thus, the versatility of case studies allows researchers to approach their topic from different angles, offering multiple ways to uncover and interpret the data .

The impact of case studies on knowledge development

Case studies play a significant role in knowledge development across various disciplines. Analysis of cases provides an avenue for researchers to explore phenomena within their context based on the collected data.

This can result in the production of rich, practical insights that can be instrumental in both theory-building and practice. Case studies allow researchers to delve into the intricacies and complexities of real-life situations, uncovering insights that might otherwise remain hidden.

Types of case studies

In qualitative research , a case study is not a one-size-fits-all approach. Depending on the nature of the research question and the specific objectives of the study, researchers might choose to use different types of case studies. These types differ in their focus, methodology, and the level of detail they provide about the phenomenon under investigation.

Understanding these types is crucial for selecting the most appropriate approach for your research project and effectively achieving your research goals. Let's briefly look at the main types of case studies.

Exploratory case studies

Exploratory case studies are typically conducted to develop a theory or framework around an understudied phenomenon. They can also serve as a precursor to a larger-scale research project. Exploratory case studies are useful when a researcher wants to identify the key issues or questions which can spur more extensive study or be used to develop propositions for further research. These case studies are characterized by flexibility, allowing researchers to explore various aspects of a phenomenon as they emerge, which can also form the foundation for subsequent studies.

Descriptive case studies

Descriptive case studies aim to provide a complete and accurate representation of a phenomenon or event within its context. These case studies are often based on an established theoretical framework, which guides how data is collected and analyzed. The researcher is concerned with describing the phenomenon in detail, as it occurs naturally, without trying to influence or manipulate it.

Explanatory case studies

Explanatory case studies are focused on explanation - they seek to clarify how or why certain phenomena occur. Often used in complex, real-life situations, they can be particularly valuable in clarifying causal relationships among concepts and understanding the interplay between different factors within a specific context.

Intrinsic, instrumental, and collective case studies

These three categories of case studies focus on the nature and purpose of the study. An intrinsic case study is conducted when a researcher has an inherent interest in the case itself. Instrumental case studies are employed when the case is used to provide insight into a particular issue or phenomenon. A collective case study, on the other hand, involves studying multiple cases simultaneously to investigate some general phenomena.

Each type of case study serves a different purpose and has its own strengths and challenges. The selection of the type should be guided by the research question and objectives, as well as the context and constraints of the research.

The flexibility, depth, and contextual richness offered by case studies make this approach an excellent research method for various fields of study. They enable researchers to investigate real-world phenomena within their specific contexts, capturing nuances that other research methods might miss. Across numerous fields, case studies provide valuable insights into complex issues.

Critical information systems research

Case studies provide a detailed understanding of the role and impact of information systems in different contexts. They offer a platform to explore how information systems are designed, implemented, and used and how they interact with various social, economic, and political factors. Case studies in this field often focus on examining the intricate relationship between technology, organizational processes, and user behavior, helping to uncover insights that can inform better system design and implementation.

Health research

Health research is another field where case studies are highly valuable. They offer a way to explore patient experiences, healthcare delivery processes, and the impact of various interventions in a real-world context.

Case studies can provide a deep understanding of a patient's journey, giving insights into the intricacies of disease progression, treatment effects, and the psychosocial aspects of health and illness.

Asthma research studies

Specifically within medical research, studies on asthma often employ case studies to explore the individual and environmental factors that influence asthma development, management, and outcomes. A case study can provide rich, detailed data about individual patients' experiences, from the triggers and symptoms they experience to the effectiveness of various management strategies. This can be crucial for developing patient-centered asthma care approaches.

Other fields

Apart from the fields mentioned, case studies are also extensively used in business and management research, education research, and political sciences, among many others. They provide an opportunity to delve into the intricacies of real-world situations, allowing for a comprehensive understanding of various phenomena.

Case studies, with their depth and contextual focus, offer unique insights across these varied fields. They allow researchers to illuminate the complexities of real-life situations, contributing to both theory and practice.

Whatever field you're in, ATLAS.ti puts your data to work for you

Download a free trial of ATLAS.ti to turn your data into insights.

Understanding the key elements of case study design is crucial for conducting rigorous and impactful case study research. A well-structured design guides the researcher through the process, ensuring that the study is methodologically sound and its findings are reliable and valid. The main elements of case study design include the research question , propositions, units of analysis, and the logic linking the data to the propositions.

The research question is the foundation of any research study. A good research question guides the direction of the study and informs the selection of the case, the methods of collecting data, and the analysis techniques. A well-formulated research question in case study research is typically clear, focused, and complex enough to merit further detailed examination of the relevant case(s).

Propositions

Propositions, though not necessary in every case study, provide a direction by stating what we might expect to find in the data collected. They guide how data is collected and analyzed by helping researchers focus on specific aspects of the case. They are particularly important in explanatory case studies, which seek to understand the relationships among concepts within the studied phenomenon.

Units of analysis

The unit of analysis refers to the case, or the main entity or entities that are being analyzed in the study. In case study research, the unit of analysis can be an individual, a group, an organization, a decision, an event, or even a time period. It's crucial to clearly define the unit of analysis, as it shapes the qualitative data analysis process by allowing the researcher to analyze a particular case and synthesize analysis across multiple case studies to draw conclusions.

Argumentation

This refers to the inferential model that allows researchers to draw conclusions from the data. The researcher needs to ensure that there is a clear link between the data, the propositions (if any), and the conclusions drawn. This argumentation is what enables the researcher to make valid and credible inferences about the phenomenon under study.

Understanding and carefully considering these elements in the design phase of a case study can significantly enhance the quality of the research. It can help ensure that the study is methodologically sound and its findings contribute meaningful insights about the case.

Ready to jumpstart your research with ATLAS.ti?

Conceptualize your research project with our intuitive data analysis interface. Download a free trial today.

Conducting a case study involves several steps, from defining the research question and selecting the case to collecting and analyzing data . This section outlines these key stages, providing a practical guide on how to conduct case study research.

Defining the research question

The first step in case study research is defining a clear, focused research question. This question should guide the entire research process, from case selection to analysis. It's crucial to ensure that the research question is suitable for a case study approach. Typically, such questions are exploratory or descriptive in nature and focus on understanding a phenomenon within its real-life context.

Selecting and defining the case

The selection of the case should be based on the research question and the objectives of the study. It involves choosing a unique example or a set of examples that provide rich, in-depth data about the phenomenon under investigation. After selecting the case, it's crucial to define it clearly, setting the boundaries of the case, including the time period and the specific context.

Previous research can help guide the case study design. When considering a case study, an example of a case could be taken from previous case study research and used to define cases in a new research inquiry. Considering recently published examples can help understand how to select and define cases effectively.

Developing a detailed case study protocol

A case study protocol outlines the procedures and general rules to be followed during the case study. This includes the data collection methods to be used, the sources of data, and the procedures for analysis. Having a detailed case study protocol ensures consistency and reliability in the study.

The protocol should also consider how to work with the people involved in the research context to grant the research team access to collecting data. As mentioned in previous sections of this guide, establishing rapport is an essential component of qualitative research as it shapes the overall potential for collecting and analyzing data.

Collecting data

Gathering data in case study research often involves multiple sources of evidence, including documents, archival records, interviews, observations, and physical artifacts. This allows for a comprehensive understanding of the case. The process for gathering data should be systematic and carefully documented to ensure the reliability and validity of the study.

Analyzing and interpreting data

The next step is analyzing the data. This involves organizing the data , categorizing it into themes or patterns , and interpreting these patterns to answer the research question. The analysis might also involve comparing the findings with prior research or theoretical propositions.

Writing the case study report

The final step is writing the case study report . This should provide a detailed description of the case, the data, the analysis process, and the findings. The report should be clear, organized, and carefully written to ensure that the reader can understand the case and the conclusions drawn from it.

Each of these steps is crucial in ensuring that the case study research is rigorous, reliable, and provides valuable insights about the case.

The type, depth, and quality of data in your study can significantly influence the validity and utility of the study. In case study research, data is usually collected from multiple sources to provide a comprehensive and nuanced understanding of the case. This section will outline the various methods of collecting data used in case study research and discuss considerations for ensuring the quality of the data.

Interviews are a common method of gathering data in case study research. They can provide rich, in-depth data about the perspectives, experiences, and interpretations of the individuals involved in the case. Interviews can be structured , semi-structured , or unstructured , depending on the research question and the degree of flexibility needed.

Observations

Observations involve the researcher observing the case in its natural setting, providing first-hand information about the case and its context. Observations can provide data that might not be revealed in interviews or documents, such as non-verbal cues or contextual information.

Documents and artifacts

Documents and archival records provide a valuable source of data in case study research. They can include reports, letters, memos, meeting minutes, email correspondence, and various public and private documents related to the case.

These records can provide historical context, corroborate evidence from other sources, and offer insights into the case that might not be apparent from interviews or observations.

Physical artifacts refer to any physical evidence related to the case, such as tools, products, or physical environments. These artifacts can provide tangible insights into the case, complementing the data gathered from other sources.

Ensuring the quality of data collection

Determining the quality of data in case study research requires careful planning and execution. It's crucial to ensure that the data is reliable, accurate, and relevant to the research question. This involves selecting appropriate methods of collecting data, properly training interviewers or observers, and systematically recording and storing the data. It also includes considering ethical issues related to collecting and handling data, such as obtaining informed consent and ensuring the privacy and confidentiality of the participants.

Data analysis

Analyzing case study research involves making sense of the rich, detailed data to answer the research question. This process can be challenging due to the volume and complexity of case study data. However, a systematic and rigorous approach to analysis can ensure that the findings are credible and meaningful. This section outlines the main steps and considerations in analyzing data in case study research.

Organizing the data

The first step in the analysis is organizing the data. This involves sorting the data into manageable sections, often according to the data source or the theme. This step can also involve transcribing interviews, digitizing physical artifacts, or organizing observational data.

Categorizing and coding the data

Once the data is organized, the next step is to categorize or code the data. This involves identifying common themes, patterns, or concepts in the data and assigning codes to relevant data segments. Coding can be done manually or with the help of software tools, and in either case, qualitative analysis software can greatly facilitate the entire coding process. Coding helps to reduce the data to a set of themes or categories that can be more easily analyzed.

Identifying patterns and themes

After coding the data, the researcher looks for patterns or themes in the coded data. This involves comparing and contrasting the codes and looking for relationships or patterns among them. The identified patterns and themes should help answer the research question.

Interpreting the data

Once patterns and themes have been identified, the next step is to interpret these findings. This involves explaining what the patterns or themes mean in the context of the research question and the case. This interpretation should be grounded in the data, but it can also involve drawing on theoretical concepts or prior research.

Verification of the data

The last step in the analysis is verification. This involves checking the accuracy and consistency of the analysis process and confirming that the findings are supported by the data. This can involve re-checking the original data, checking the consistency of codes, or seeking feedback from research participants or peers.

Like any research method , case study research has its strengths and limitations. Researchers must be aware of these, as they can influence the design, conduct, and interpretation of the study.

Understanding the strengths and limitations of case study research can also guide researchers in deciding whether this approach is suitable for their research question . This section outlines some of the key strengths and limitations of case study research.

Benefits include the following:

• Rich, detailed data: One of the main strengths of case study research is that it can generate rich, detailed data about the case. This can provide a deep understanding of the case and its context, which can be valuable in exploring complex phenomena.
• Flexibility: Case study research is flexible in terms of design , data collection , and analysis . A sufficient degree of flexibility allows the researcher to adapt the study according to the case and the emerging findings.
• Real-world context: Case study research involves studying the case in its real-world context, which can provide valuable insights into the interplay between the case and its context.
• Multiple sources of evidence: Case study research often involves collecting data from multiple sources , which can enhance the robustness and validity of the findings.

On the other hand, researchers should consider the following limitations:

• Generalizability: A common criticism of case study research is that its findings might not be generalizable to other cases due to the specificity and uniqueness of each case.
• Time and resource intensive: Case study research can be time and resource intensive due to the depth of the investigation and the amount of collected data.
• Complexity of analysis: The rich, detailed data generated in case study research can make analyzing the data challenging.
• Subjectivity: Given the nature of case study research, there may be a higher degree of subjectivity in interpreting the data , so researchers need to reflect on this and transparently convey to audiences how the research was conducted.

Being aware of these strengths and limitations can help researchers design and conduct case study research effectively and interpret and report the findings appropriately.

Ready to analyze your data with ATLAS.ti?

See how our intuitive software can draw key insights from your data with a free trial today.

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Acquisition
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Religion
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business Strategy
• Business History
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Systems
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Politics and Law
• Public Policy
• Public Administration
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

A newer edition of this book is available.

• < Previous chapter
• Next chapter >

22 Case Study Research: In-Depth Understanding in Context

Helen Simons, School of Education, University of Southampton

• Published: 01 July 2014
• Cite Icon Cite
• Permissions Icon Permissions

This chapter explores case study as a major approach to research and evaluation. After first noting various contexts in which case studies are commonly used, the chapter focuses on case study research directly Strengths and potential problematic issues are outlined and then key phases of the process. The chapter emphasizes how important it is to design the case, to collect and interpret data in ways that highlight the qualitative, to have an ethical practice that values multiple perspectives and political interests, and to report creatively to facilitate use in policy making and practice. Finally, it explores how to generalize from the single case. Concluding questions center on the need to think more imaginatively about design and the range of methods and forms of reporting requiredto persuade audiences to value qualitative ways of knowing in case study research.

Introduction

This chapter explores case study as a major approach to research and evaluation using primarily qualitative methods, as well as documentary sources, contemporaneous or historical. However, this is not the only way in which case study can be conceived. No one has a monopoly on the term. While sharing a focus on the singular in a particular context, case study has a wide variety of uses, not all associated with research. A case study, in common parlance, documents a particular situation or event in detail in a specific sociopolitical context. The particular can be a person, a classroom, an institution, a program, or a policy. Below I identify different ways in which case study is used before focusing on qualitative case study research in particular. However, first I wish to indicate how I came to advocate and practice this form of research. Origins, context, and opportunity often shape the research processes we endorse. It is helpful for the reader, I think, to know how I came to the perspective I hold.

The Beginnings

I first came to appreciate and enjoy the virtues of case study research when I entered the field of curriculum evaluation and research in the 1970s. The dominant research paradigm for educational research at that time was experimental or quasi- experimental, cost-benefit, or systems analysis, and the dominant curriculum model was aims and objectives ( House, 1993 ). The field was dominated, in effect, by a psychometric view of research in which quantitative methods were preeminent. But the innovative projects we were asked to evaluate (predominantly, but not exclusively, in the humanities) were not amenable to such methodologies. The projects were challenging to the status quo of institutions, involved people interpreting the policy and programs, were implemented differently in different contexts and regions, and had many unexpected effects.

We had no choice but to seek other ways to evaluate these complex programs, and case study was the methodology we found ourselves exploring, in order to understand how the projects were being implemented, why they had positive effects in some regions of the country and not others, and what the outcomes meant in different sociopolitical and cultural contexts. What better way to do this than to talk with people to see how they interpreted the “new” curriculum; to watch how teachers and students put it into practice; to document transactions, outcomes, and unexpected consequences; and to interpret all in the specific context of the case ( Simons, 1971 , 1987 , pp. 55–89). From this point on and in further studies, case study in educational research and evaluation came to be a major methodology for understanding complex educational and social programs. It also extended to other practice professions, such as nursing, health, and social care ( Zucker, 2001 ; Greenhalgh & Worrall, 1997 ; Shaw & Gould, 2001 ). For further details of the evolution of the case study approach and qualitative methodologies in evaluation, see House, 1993 , pp. 2–3; Greene, 2000 ; Simons, 2009 , pp. 14–18; Simons & McCormack, 2007 , pp. 292–311).

This was not exactly the beginning of case study, of course. It has a long history in many disciplines ( Simons, 1980; Ragin, 1992; Gomm, Hammersley, & Foster, 2004 ; Platt, 2007 ), many aspects of which form part of case study practice to this day. But its evolution in the context just described was a major move in the contemporary evolution of the logic of evaluative inquiry ( House, 1980 ). It also coincided with movement toward the qualitative in other disciplines, such as sociology and psychology. This was all part of what Denzin & Lincoln (1994) termed “a quiet methodological revolution” (p. ix) in qualitative inquiry that had been evolving over the course of forty years.

There is a further reason why I continue to advocate and practice case study research and evaluation to this day and that is my personal predilection for trying to understand and represent complexity, for puzzling through the ambiguities that exist in many contexts and programs and for presenting and negotiating different values and interests in fair and just ways.

Put more simply, I like interacting with people, listening to their stories, trials and tribulations—giving them a voice in understanding the contexts and projects with which they are involved, and finding ways to share these with a range of audiences. In other words, the move toward case study methodology described here suited my preference for how I learn.

Concepts and Purposes of Case Study

Before exploring case study as it has come to be established in educational research and evaluation over the past forty years, I wish to acknowledge other uses of case study. More often than not, these relate to purpose, and appropriately so in their different contexts, but many do not have a research intention. For a study to count as research, it would need to be a systematic investigation generating evidence that leads to “new” knowledge that is made public and open to scrutiny. There are many ways to conduct research stemming from different traditions and disciplines, but they all, in different ways, involve these characteristics.

Everyday Usage: Stories We Tell

The most common of these uses of case study is the everyday reference to a person, an anecdote or story illustrative of a particular incident, event, or experience of that person. It is often a short, reported account commonly seen in journalism but also in books exploring a phenomenon, such as recovery from serious accidents or tragedies, where the author chooses to illustrate the story or argument with a “lived” example. This is sometimes written by the author and sometimes by the person whose tale it is. “Let me share with you a story,” is a phrase frequently heard

The spirit behind this common usage and its power to connect can be seen in a report by Tim Adams of the London Olympics opening ceremony’s dramatization by Danny Boyle.

It was the point when we suddenly collectively wised up to the idea that what we are about to receive over the next two weeks was not only about “legacy collateral” and “targeted deliverables,” not about G4S failings and traffic lanes and branding opportunities, but about the second-by-second possibilities of human endeavour and spirit and communality, enacted in multiple places and all at the same time. Stories in other words. ( Adams, 2012 )

This was a collective story, of course, not an individual one, but it does convey some of the major characteristics of case study—that richness of detail, time, place, multiple happenings and experiences—that are also manifest in case study research, although carefully evidenced in the latter instance. We can see from this common usage how people have come to associate case study with story. I return to this thread in the reporting section.

Professions Individual Cases

In professional settings, in health and social care, case studies, often called case histories , are used to accurately record a person’s health or social care history and his or her current symptoms, experience, and treatment. These case histories include facts but also judgments and observations about the person’s reaction to situations or medication. Usually these are confidential. Not dissimilar is the detailed documentation of a case in law, often termed a case precedent when referred to in a court case to support an argument being made. However in law there is a difference in that such case precedents are publicly documented.

Case Studies in Teaching

Exemplars of practice.

In education, but also in health and social care training contexts, case studies have long been used as exemplars of practice. These are brief descriptions with some detail of a person or project’s experience in an area of practice. Though frequently reported accounts, they are based on a person’s experience and sometimes on previous research.

Case scenarios

Management studies are a further context in which case studies are often used. Here, the case is more like a scenario outlining a particular problem situation for the management student to resolve. These scenarios may be based on research but frequently are hypothetical situations used to raise issues for discussion and resolution. What distinguishes these case scenarios and the case exemplars in education from case study research is the intention to use them for teaching purposes.

Country Case Studies

Then there are case studies of programs, projects, and even countries, as in international development, where a whole-country study might be termed a case study or, in the context of the Organization for Economic Co-operation and Development (OECD), where an exploration is conducted of the state of the art of a subject, such as education or environmental science in one or several countries. This may be a contemporaneous study and/or what transpired in a program over a period of time. Such studies often do have a research base but frequently are reported accounts that do not detail the design, methodology, and analysis of the case, as a research case study would do, or report in ways that give readers a vicarious experience of what it was like to be there. Such case studies tend to be more knowledge and information-focused than experiential.

Case Study as History

Closer to a research context is case study as history—what transpired at a certain time in a certain place. This is likely to be supported by documentary evidence but not primary data gathering unless it is an oral history. In education, in the late 1970s, Stenhouse (1978) experimented with a case study archive. Using contemporaneous data gathering, primarily through interviewing, he envisaged this database, which he termed a “case record,” forming an archive from which different individuals,, at some later date, could write a “case study.” This approach uses case study as a documentary source to begin to generate a history of education, as the subtitle of Stenhouse’s 1978 paper indicates “Towards a contemporary history of education.”

Case Study Research

From here on, my focus is on case study research per se, adopting for this purpose the following definition:

Case study is an in-depth exploration from multiple perspectives of the complexity and uniqueness of a particular project, policy, institution or system in a “real-life” context. It is research based, inclusive of different methods and is evidence-led. ( Simons, 2009 , p. 21).

For further related definitions of case study, see Stake (1995) , Merriam (1998), and Chadderton & Torrance (2011) . And for definitions from a slightly different perspective, see Yin (2004) and Thomas (2011a) .

Not Defined by Method or Perspective

The inclusion of different methods in the definition quoted above definition signals that case study research is not defined by methodology or method. What defines case study is its singularity and the concept and boundary of the case. It is theoretically possible to conduct a case study using primarily quantitative data if this is the best way of providing evidence to inform the issues the case is exploring. It is equally possible to conduct case study that is mainly qualitative, to engage people with the experience of the case or to provide a rich portrayal of an event, project, or program.

Or one can design the case using mixed methods. This increases the options for learning from different ways of knowing and is sometimes preferred by stakeholders who believe it provides a firmer basis for informing policy. This is not necessarily the case but is beyond the scope of this chapter to explore. For further discussion of the complexities of mixing methods and the virtue of using qualitative methods and case study in a mixed method design, see Greene (2007) .

Case study research may also be conducted from different standpoints—realist, interpretivist, or constructivist, for example. My perspective falls within a constructivist, interpretivist framework. What interests me is how I and those in the case perceive and interpret what we find and how we construct or co-construct understandings of the case. This not only suits my predilection for how I see the world, but also my preferred phenomenological approach to interviewing and curiosity about people and how they act in social and professional life.

Qualitative Case Study Research

Qualitative case study research shares many characteristics with other forms of qualitative research, such as narrative, oral history, life history, ethnography, in-depth interview, and observational studies that utilize qualitative methods. However, its focus, purpose, and origins, in educational research at least, are a little different.

The focus is clearly the study of the singular. The purpose is to portray an in-depth view of the quality and complexity of social/educational programs or policies as they are implemented in specific sociopolitical contexts. What makes it qualitative is its emphasis on subjective ways of knowing, particularly the experiential, practical, and presentational rather than the propositional ( Heron, 1992 , 1999 ) to comprehend and communicate what transpired in the case.

Characteristic Features and Advantages

Case study research is not method dependent, as noted earlier, nor is it constrained by resources or time. Although it can be conducted over several years, which provides an opportunity to explore the process of change and explain how and why things happened, it can equally be carried out contemporaneously in a few days, weeks, or months. This flexibility is extremely useful in many contexts, particularly when a change in policy or unforeseen issues in the field require modifying the design.

Flexibility extends to reporting. The case can be written up in different lengths and forms to meet different audience needs and to maximize use (see the section on Reporting). Using the natural language of participants and familiar methods (like interview, observation, oral history) also enables participants to engage in the research process, thereby contributing significantly to the generation of knowledge of the case. As I have indicated elsewhere ( Simons, 2009 ), “This is both a political and epistemological point. It signals a potential shift in the power base of who controls knowledge and recognizes the importance of co-constructing perceived reality through the relationships and joint understandings we create in the field” (p. 23).

Possible Disadvantages

If one is an advocate, identifying advantages of a research approach is easier than pointing out its disadvantages, something detractors are quite keen to do anyway! But no approach is perfect, and here are some of the issues that often trouble people about case study research. The “sample of one” is an obvious issue that worries those convinced that only large samples can constitute valid research and especially if this is to inform policy. Understanding complexity in depth may not be a sufficient counterargument, and I suspect there is little point in trying to persuade otherwise For frequently, this perception is one of epistemological and methodological, if not ideological, preference.

However, there are some genuine concerns that many case researchers face: the difficulty of processing a mass of data; of “telling the truth” in contexts where people may be identifiable; personal involvement, when the researcher is the main instrument of data gathering; and writing reports that are data-based, yet readable in style and length. But one issue that concerns advocates and nonadvocates alike is how inferences are drawn from the single case.

Answers to some of these issues are covered in the sections that follow. Whether they convince may again be a question of preference. However, it is worth noting here that I do not think we should seek to justify these concerns in terms identified by other methodologies. Many of them are intrinsic to the nature and strength of qualitative case study research.

Subjectivity, for instance, both of participants and researcher is inevitable, as it is in many other qualitative methodologies. This is often the basis on which we act. Rather than see this as bias or something to counter, it is an intelligence that is essential to understanding and interpreting the experience of participants and stakeholders. Such subjectivity needs to be disciplined, of course, through procedures that examine both the validity of individuals’ representations of “their truth”, and demonstrate how the researcher took a reflexive approach to monitoring how his or her own values and predilections may have unduly influenced the data.

Types of Case Study

There are numerous types of case study, too many to categorize, I think, as there are overlaps between them. However, attempts have been made to do this and, for those who value typologies, I refer them to Bassey (1999) and, for a more extended typology, to Thomas (2011b) . A slightly different approach is taken by Gomm, Hammersley, and Foster (2004) in annotating the different emphases in major texts on case study. What I prefer to do here is to highlight a few familiar types to focus the discussion that follows on the practice of case study research.

Stake (1995) offers a threefold distinction that is helpful when it comes to practice, he says, because it influences the methods we choose to gather data (p. 4). He distinguishes between an intrinsic case study , one that is studied to learn about the particular case itself and an instrumental case study , in which we choose a case to gain insight into a particular issue (i.e., the case is instrumental to understanding something else; p. 3). The collective case study is what its name suggests: an extension of the instrumental to several cases.

Theory-led or theory-generated case study is similarly self-explanatory, the first starting from a specific theory that is tested through the case; the second constructing a theory through interpretation of data generated in the case. In other words, one ends rather than begins with a theory. In qualitative case study research, this is the more familiar route. The theory of the case becomes the argument or story you will tell.

Evaluation case study requires a slightly longer description as this is my context of practice, one which has influenced the way I conduct case study and what I choose to emphasize in this chapter. An evaluation case study has three essential features: to determine the value of the case, to include and balance different interests and values, and to report findings to a range of stakeholders in ways that they can use. The reasons for this may be found in the interlude that follows, which offers a brief characterization of the social and ethical practice of evaluation and why qualitative methods are so important in this practice.

Interlude: Social and Ethical Practice of Evaluation

Evaluation is a social practice that documents, portrays, and seeks to understand the value of a particular project, program, or policy. This can be determined by different evaluation methodologies, of course. But the value of qualitative case study is that it is possible to discern this value without decontextualizing the data. While the focus of the case is usually a project, program, policy, or some unit within, studies of key individuals, what I term case profiles , may be embedded within the overall case. In some instances, these profiles, or even shorter cameos of individuals, may be quite prominent. For it is through the perceptions, interpretations, and interactions of people that we learn how policies and programs are enacted ( Kushner, 2000 , p. 12). The program is still the main focus of analysis, but, in exploring how individuals play out their different roles in the program, we get closer to the actual experience and meaning of the program in practice.

Case study evaluation is often commissioned from an external source (government department or other agency) keen to know the worth of publicly funded programs and policies to inform future decision making. It needs to be responsive to issues or questions identified by stakeholders, who often have different values and interests in the expected outcomes and appreciate different perspectives of the program in action. The context also is often highly politicized, and interests can conflict. The task of the evaluator in such situations becomes one of including and balancing all interests and values in the program fairly and justly.

This is an inherently political process and requires an ethical practice that offers participants some protection over the personal data they give as part of the research and agreed audiences access to the findings, presented in ways they can understand. Negotiating what information becomes public can be quite difficult in singular settings where people are identifiable and intricate or problematic transactions have been documented. The consequences that ensue from making knowledge public that hitherto was private may be considerable for those in the case. It may also be difficult to portray some of the contextual detail that would enhance understanding for readers.

The ethical stance that underpins the case study research and evaluation I conduct stems from a theory of ethics that emphasizes the centrality of relationships in the specific context and the consequences for individuals, while remaining aware of the research imperative to publicly report. It is essentially an independent democratic process based on the concepts of fairness and justice, in which confidentiality, negotiation, and accessibility are key principles ( MacDonald, 1976 ; Simons, 2009 , pp. 96–111; and Simons 2010 ). The principles are translated into specific procedures to guide the collection, validation, and dissemination of data in the field. These include:

engaging participants and stakeholders in identifying issues to explore and sometimes also in interpreting the data;

documenting how different people interpret and value the program;

negotiating what data becomes public respecting both the individual’s “right to privacy” and the public’s “right to know”;

offering participants opportunities to check how their data are used in the context of reporting;

reporting in language and forms accessible to a wide range of audiences;

disseminating to audiences within and beyond the case.

For further discussion of the ethics of democratic case study evaluation and examples of their use in practice, see Simons (2000 , 2006 , 2009 , chapter 6, 2010 ).

Designing Case Study Research

Design issues in case study sometimes take second place to those of data gathering, the more exciting task perhaps in starting research. However, it is critical to consider the design at the outset, even if changes are required in practice due to the reality of what is encountered in the field. In this sense, the design of case study is emergent, rather than preordinate, shaped and reshaped as understanding of the significance of foreshadowed issues emerges and more are discovered.

Before entering the field, there are a myriad of planning issues to think about related to stakeholders, participants, and audiences. These include whose values matter, whether to engage them in data gathering and interpretation, the style of reporting appropriate for each, and the ethical guidelines that will underpin data collection and reporting. However, here I emphasize only three: the broad focus of the study, what the case is a case of, and framing questions/issues. These are steps often ignored in an enthusiasm to gather data, resulting in a case study that claims to be research but lacks the basic principles required for generation of valid, public knowledge.

Conceptualize the Topic

First, it is important that the topic of the research is conceptualized in a way that it can be researched (i.e., it is not too wide). This seems an obvious point to make, but failure to think through precisely what it is about your research topic you wish to investigate will have a knock-on effect on the framing of the case, data gathering, and interpretation and may lead, in some instances, to not gathering or analyzing data that actually informs the topic. Further conceptualization or reconceptualization may be necessary as the study proceeds, but it is critical to have a clear focus at the outset.

What Constitutes the Case

Second, I think it is important to decide what would constitute the case (i.e., what it is a case of) and where the boundaries of this lie. This often proves more difficult than first appears. And sometimes, partly because of the semifluid nature of the way the case evolves, it is only possible to finally establish what the case is a case of at the end. Nevertheless, it is useful to identify what the case and its boundaries are at the outset to help focus data collection while maintaining an awareness that these may shift. This is emergent design in action.

In deciding the boundary of the case, there are several factors to bear in mind. Is it bounded by an institution or a unit within an institution, by people within an institution, by region, or by project, program or policy,? If we take a school as an example, the case could be comprised of the principal, teachers, and students, or the boundary could be extended to the cleaners, the caretaker, the receptionist, people who often know a great deal about the subnorms and culture of the institution.

If the case is a policy or particular parameter of a policy, the considerations may be slightly different. People will still be paramount—those who generated the policy and those who implemented it—but there is likely also to be a political culture surrounding the policy that had an influence on the way the policy evolved. Would this be part of the case?

Whatever boundary is chosen, this may change in the course of conducting the study when issues arise that can only be understood by going to another level. What transpires in a classroom, for example, if this is the case, is often partly dependent on the support of the school leadership and culture of the institution and this, in turn, to some extent is dependent on what resources are allocated from the local education administration. Much like a series of Russian dolls, one context inside the other.

Unit of analysis

Thinking about what would constitute the unit of analysis— a classroom, an institution, a program, a region—may help in setting the boundaries of the case, and it will certainly help when it comes to analysis. But this is a slightly different issue from deciding what the case is a case of. Taking a health example, the case may be palliative care support, but the unit of analysis the palliative care ward or wards. If you took the palliative care ward as the unit of analysis this would be as much about how palliative care was exercised in this or that ward than issues about palliative care support in general. In other words, you would need to have specific information and context about how this ward was structured and managed to understand how palliative care was conducted in this particular ward. Here, as in the school example above, you would need to consider which of the many people who populate the ward form part of the case—nurses, interns, or doctors only, or does it extend to patients, cleaners, nurse aides, and medical students?

Framing Questions and Issues

The third most important consideration is how to frame the study, and you are likely to do this once you have selected the site or sites for study. There are at least four approaches. You could start with precise questions, foreshadowed issues ( Smith & Pohland, 1974 ), theories, or a program logic. To some extent, your choice will be dictated by the type of case you have chosen, but also by your personal preference for how to conduct it—in either a structured or open way.

Initial questions give structure; foreshadowed issues more freedom to explore. In qualitative case study, foreshadowed issues are more common, allowing scope for issues to change as the study evolves, guided by participants’ perspectives and events in the field. With this perspective, it is more likely that you will generate a theory of the case toward the end, through your interpretation and analysis.

If you are conducting an instrumental case study, staying close to the questions or foreshadowed issues is necessary to be sure you gain data that will illuminate the central focus of the study. This is critical if you are exploring issues across several cases, although it is possible to do a cross-case analysis from cases that have each followed a different route to discovering significant issues.

Opting to start with a theoretical framework provides a basis for formulating questions and issues, but it can also constrain the study to only those questions/issues that fit the framework. The same is true with using program logic to frame the case. This is an approach frequently adopted in evaluation case study where the evaluator, individually or with stakeholders, examines how the aims and objectives of the program relate to the activities designed to promote it and the outcomes and impacts expected. It provides direction, although it can lead to simply confirming what was anticipated, rather than documenting what transpired in the case.

Whichever approach you choose to frame the case, it is useful to think about the rationale or theory for each question and what methods would best enable you to gain an understanding of them. This will not only start a reflexive process of examining your choices—an important aspect of the process of data gathering and interpretation—it will also aid analysis and interpretation further down the track.

Methodology and Methods

Qualitative case study research, as already noted, appeals to subjective ways of knowing and to a primarily qualitative methodology, that captures experiential understanding ( Stake, 2010 , pp. 56–70). It follows that the main methods of data gathering to access this way of knowing will be qualitative. Interviewing, observation, and document analysis are the primary three, often supported by critical incidents, focus groups, cameos, vignettes, diaries/journals, and photographs. Before gathering any primary data, however, it is useful to search relevant existing sources (written or visual) to learn about the antecedents and context of a project, program, or policy as a backdrop to the case. This can sharpen framing questions, avoid unnecessary data gathering, and shorten the time needed in the field.

Given that there are excellent texts on qualitative methods (see, for example, Denzin & Lincoln, 1994 ; Seale, 1999 ; Silverman, 2000 , 2004 ), I will not discuss all potential relevant methods here, but simply focus on the qualities of the primary methods that are particularly appropriate for case study research.

Primary Qualitative Data Gathering Methods

Interviewing.

The most effective style of interviewing in qualitative case study research to gain in-depth data, document multiple perspectives and experiences and explore contested issues is the unstructured interview, active listening and open questioning are paramount, whatever prequestions or foreshadowed issues have been identified. This can include photographs—a useful starting point with certain cultural groups and the less articulate, to encourage them to tell their story through connecting or identifying with something in the image.

The flexibility of unstructured interviewing has three further advantages for understanding participants’ experiences. First, through questioning, probing, listening, and, above all, paying attention to the silences and what they mean, you can get closer to the meaning of participants’ experiences. It is not always what they say.

Second, unstructured interviewing is useful for engaging participants in the process of research. Instead of starting with questions and issues, invite participants to tell their stories or reflect on specific issues, to conduct their own self-evaluative interview, in fact. Not only will they contribute their particular perspective to the case, they will also learn about themselves, thereby making the process of research educative for them as well as for the audiences of the research.

Third, the open-endedness of this style of interviewing has the potential for creating a dialogue between participants and the researcher and between the researcher and the public, if enough of the dialogue is retained in the publication ( Bellah, Madsen, Sullivan, Swidler, & Tipton, 1985 ).

Observations

Observations in case study research are likely to be close-up descriptions of events, activities, and incidents that detail what happens in a particular context. They will record time, place, specific incidents, transactions, and dialogue, and note characteristics of the setting and of people in it without preconceived categories or judgment. No description is devoid of some judgment in selection, of course, but, on the whole, the intent is to describe the scene or event “as it is,” providing a rich, textured description to give readers a sense of what it was like to be there or provide a basis for later interpretation.

Take the following excerpt from a study of the West Bromwich Operatic Society. It is the first night of a new production, The Producers , by this amateur operatic society. This brief excerpt is from a much longer observation of the overture to the first evening’s performance, detailing exactly what the production is, where it is, and why there is such a tremendous sense of atmosphere and expectation surrounding the event. Space prevents including the whole observation, but I hope you can get a glimmer of the passion and excitement that precedes the performance:

Birmingham, late November, 2011, early evening.... Bars and restaurants spruce up for the evening’s trade. There is a chill in the air but the party season is just starting....

A few hundred yards away, past streaming traffic on Suffolk Street, Queensway, an audience is gathering at the New Alexandra Theatre. The foyer windows shine in the orange sodium night. Above each one is the rubric: WORLD CLASS THEATRE.

Inside the preparatory rituals are being observed; sweets chosen, interval drinks ordered and programmes bought. People swap news and titbits about the production.... The bubble of anticipation grows as the 5-minute warning sounds. People make their way to the auditorium. There have been so many nights like this in the past 110 years since a man named William Coutts invested £10,000 to build this palace of dreams.... So many fantasies have been played under this arch: melodramas and pantomimes, musicals and variety.... So many audiences, settling down in their tip-up seats, wanting to be transported away from work, from ordinariness and private troubles.... The dimming lights act like a mother’s hush. You could touch the silence. Boinnng! A spongy thump on a bass drum, and the horns pipe up that catchy, irrepressible, tasteless tune and already you’re singing under your breath, ‘Springtime for Hitler and Germany....’ The orchestra is out of sight in the pit. There’s just the velvet curtain to watch as your fingers tap along. What’s waiting behind? Then it starts it to move. Opening night.... It’s opening night! ( Matarasso, 2012 , pp. 1–2)

For another and different example—a narrative observation of an everyday but unique incident that details date, time, place, and experience—see Simons (2009 , p. 60).

Such naturalistic observations are also useful in contexts where we cannot understand what is going on through interviewing alone—in cultures with which we are less familiar or where key actors may not share our language or have difficulty expressing it. Careful description in these situations can help identify key issues, discover the norms and values that exist in the culture, and, if sufficiently detailed, allow others to cross corroborate what significance we draw from these observations. This last point is very important to avoid the danger in observation of ascribing motivations to people and meanings to transactions.

Finally, naturalistic observations are very important in highly politicized environments, often the case in commissioned evaluation case study, where individuals in interview may try to elude the “truth” or press on you that their view is the “right” view of the situation. In these contexts, naturalistic observations not only enable you to document interactions as you perceive them, but they also provide a cross-check on the veracity of information obtained in interviews.

Document analysis

Analysis of documents, as already intimated, is useful for establishing what historical antecedents might exist to provide a springboard for contemporaneous data gathering. In most cases, existing documents are also extremely pertinent for understanding the policy context.

In a national policy case study I conducted on a major curriculum change, the importance of preexisting documentation was brought home to me sharply when certain documentation initially proved elusive to obtain. It was difficult to believe that it did not exist, as the evolution of the innovation involved several parties who had not worked together before. There was bound, I thought, to be minuted meetings sharing progress and documentation of the “new” curriculum. In the absence of some crucial documents, I began to piece together the story through interviewing. Only there were gaps, and certain issues did not make sense.

It was only when I presented two versions of what I discerned had transpired in the development of this initiative in an interim report eighteen months into the study that things started to change. Subsequent to the meeting at which the report was presented, the “missing” documents started to appear. Suddenly found. What lay behind the “missing documents,” something I suspected from what certain individuals did and did not say in interview, was a major difference of view about how the innovation evolved, who was key in the process, and whose voice was more important in the context. Political differences, in other words, that some stakeholders were trying to keep from me. The emergence of the documents enabled me to finally produce an accurate and fair account.

This is an example of the importance of having access to all relevant documents relating to a program or policy in order to study it fairly. The other major way in which document analysis is useful in case study is for understanding the values, explicit and hidden, in policy and program documents and in the organization where the program or policy is implemented. Not to be ignored as documents are photographs, and these, too, can form the basis of a cultural and value analysis of an organization ( Prosser, 2000 ).

Creative artistic approaches

Increasingly, some case study researchers are employing creative approaches associated with the arts as a means of data gathering and analysis. Artistic approaches have often been used in representing findings, but less frequently in data gathering and interpretation ( Simons & McCormack, 2007 ). A major exception is the work of Richardson (1994) , who sees the very process of writing as an interpretative act, and of Cancienne and Snowber (2003) , who argue for movement as method.

The most familiar of these creative and artistic forms are written—narratives and short stories ( Clandinin & Connelly, 2000 ; Richardson, 1994 ; Sparkes, 2002 ), poems or poetic form ( Butler-Kisber, 2010 ; Duke, 2007 ; Richardson, 1997 ; Sparkes & Douglas, 2007 ), cameos of people, or vignettes of situations. These can be written by participants or by the researcher or developed in partnership. They can also be shared with participants to further interpret the data. But photographs also have a long history in qualitative research for presenting and constructing understanding ( Butler-Kisber, 2010 ; Collier, 1967 ; Prosser, 2000 ; Rugang, 2006 ; Walker, 1993 ).

Less common are other visual forms of gathering data, such as “draw and write” ( Sewell, 2011 ), artefacts, drawings, sketches, paintings, and collages, although all forms are now on the increase. For examples of the use of collage in data gathering, see Duke (2007) and Butler-Kisber (2010) , and for charcoal drawing, Elliott (2008) .

In qualitative inquiry broadly, these creative approaches are now quite common. And in the context of arts and health in particular (see, for example, Frank, 1997 ; Liamputtong & Rumbold, 2008 ; Spouse, 2000 ), we can see how artistic approaches illuminate in-depth understanding. However, in case study research to date, I think narrative forms have tended to be most prominent.

Finally, for capturing the quality and essence of peoples’ experience, nothing could be more revealing than a recording of their voices. Video diaries—self-evaluative portrayals by individuals of their perspectives, feelings, or experience of an event or situation—are a most potent way both of gaining understanding and communicating that to others. It is rather more difficult to gain access for observational videos, but they are useful for documentation and have the potential to engage participants and stakeholders in the interpretation.

Getting It All Together

Case study is so often associated with story or with a report of some event or program that it is easy to forget that much analysis and interpretation has gone on before we reach this point. In many case study reports, this process is hidden, leaving the reader with little evidence on which to assess the validity of the findings and having to trust the one who wrote the tale.

This section briefly outlines possibilities, first, for analyzing and interpreting data, and second, for how to communicate the findings to others. However it is useful to think of these together and indeed, at the start, because decisions about how you report may influence how you choose to make sense of the data. Your choice may also vary according to the context of the study—what is expected or acceptable—and your personal predilections, whether you prefer a more rational than intuitive mode of analysis, for example, or a formal or informal style of writing up that includes images, metaphor, narratives, or poetic forms.

Analyzing and Interpreting Data

When it comes to making sense of data, I make a distinction between analysis—a formal inductive process that seeks to explain—and interpretation, a more intuitive process that gains understanding and insight from a holistic grasp of data, although these may interact and overlap at different stages.

The process, whichever emphasis you choose, is one of reducing or transforming a large amount of data to themes that can encapsulate the overarching meaning in the data. This involves sorting, refining, and refocusing data until they make sense. It starts at the beginning with preliminary hunches, sometimes called “interpretative asides” or “working hypotheses,” later moving to themes, analytic propositions, or a theory of the case.

There are many ways to conduct this process. Two strategies often employed are concept mapping —a means of representing data visually to explore links between related concepts—and progressive focusing ( Parlett & Hamilton, 1976 ), the gradual reframing of initially identified issues into themes that are then further interpreted to generate findings. Each of these strategies tends to have three stages: initial sense making, identification of themes, and examination of patterns and relationships between them.

If taking a formal analytic approach to the task, the data would likely be broken down into segments or datasets (coded and categorized) and then reordered and explored for themes, patterns, and possible propositions. If adopting a more intuitive process, you might focus on identifying insights through metaphors and images, lateral thinking, or puzzling over paradoxes and ambiguities in the data, after first immersing yourself in the total dataset, reading and re-reading interview scripts, observations and field notes to get a sense of the whole. Trying out different forms of making sense through poetry, vignettes, cameos, narratives, collages, and drawing are further creative ways to interpret data, as are photographs taken in the case arranged to explain or tell the story of the case.

Reporting Case Study Research

Narrative structure and story.

As indicated in the introduction, telling a story is often associated with case study and some think this is what a case study is. In one sense, it is and, given that story is the natural way in which we learn ( Okri, 1997 ), it is a useful framework both for gathering data and for communicating case study findings. Not any story will do however. To count as research, it must be authentic, grounded in data, interpreted and analyzed to convey the meaning of the case.

There are several senses in which story is appropriate in qualitative case study: in capturing stories participants tell, in generating a narrative structure that makes sense of the case (i.e., the story you will tell), and in deciding how you communicate this narrative (i.e., in story form). If you choose a written story form (and advice here can be sought from Harrington (2003) and Caulley (2008) ), it needs to be clearly structured, well written, and contain only the detail that is necessary to give readers the vicarious experience of what it was like in the case. If the story is to be communicated in other ways, through, for example, audio or videotape, or computer or personal interaction, the same applies, substituting visual and interpersonal skill for written.

Matching forms of reporting to audience

The art of reporting is strongly connected to usability, so forms of reporting need to connect to the audiences we hope to inform: how they learn, what kind of evidence they value, and what kind of reporting maximizes the chances they will use the findings to promote policies and programs in the interests of beneficiaries. As Okri (1997) further reminds us, the writer only does half the work; the reader does the other (p. 41).

There may be other considerations as well: how open are commissioners to receiving stories of difficulties, as well as success stories? What might they need to hear beyond what is sought in the technical brief? And through what style of reporting would you try and persuade them? If conducting noncommissioned case study research, the scope for different forms of reporting is wider. In academia, for instance, many institutions these days accept creative and artistic forms of reporting when supported by supervisors and appreciated by examiners.

Styles of Reporting

The most obvious form of reporting is linear, often starting with a short executive summary and a brief description of focus and context, followed by methodology, the case study or thematic analysis, findings, and conclusions or implications. Conclusion-led reporting is similar in terms of its formality, but simply starts the other way around. From the conclusions drawn from the analyzed data, it works backward to tell the story through narrative, verbatim, and observational data of how these conclusions were reached. Both have a strong story line. The intent is analytic and explanatory.

Quite a different approach is to engage the reader in the experience and veracity of the case. Rather like constructing a portrait or editing a documentary film, this involves the sifting, constructing, re-ordering of frames, events and episodes to tell a coherent story primarily through interview excerpts, observations, vignettes, and critical incidents that depict what transpired in the case. Interpretation is indirect through the weaving of the data. The story can start at any point provided the underlying narrative structure is maintained to establish coherence ( House, 1980 , p. 116).

Different again, and from the other end of a continuum, is a highly interpretative account that may use similar ways of presenting data but weaves a story from the outset that is highly interpretative. Engaging metaphor, images, short stories, contradictions, paradoxes, and puzzles, it is invariably interesting to read and can be most persuasive. However, the evidence is less visible and therefore less open to alternative interpretations.

Even more persuasive is a case study that uses artistic forms to communicate the story of the case. Paintings, poetic form, drawings, photography, collage, and movement can all be adopted to report findings, whether the data was acquired using these forms or by other means. The arts-based inquiry movement ( Mullen & Finley, 2003 ) has contributed hugely to the validation and legitimation of artistic and creative ways of representing qualitative research findings. The journal Qualitative Inquiry contains many good examples, but see also Liamputtong & Rumbold (2008) . Such artistic forms of representation may not be for everyone or appropriate in some contexts, but they do have the power to engage an audience and the potential to facilitate use.

Generalization in Case Study Research

One of the potential limitations of case study often proposed is that it is impossible to generalize. This is not so. However, the way in which one generalizes from a case is different from that adopted in traditional forms of social science research that utilize large samples (randomly selected) and statistical procedures and which assume regularities in the social world that allow cause and effect to be determined. In this form of research inferences from data are stated as formal propositions that apply to all in the target population. See Donmoyer (1990) for an argument on the restricted nature of this form of generalization when considering single-case studies.

Making inferences from cases with a qualitative data set arises more from a process of interpretation in context, appealing to tacit and situated understanding for acceptance of their validity. Such inferences are possible where the context and experience of the case is richly described so the reader can recognize and connect with the events and experiences portrayed. There are two ways to examine how to reach these generalized understandings. One is to generalize from the case to other cases of a similar or dissimilar nature. The other is to see what we learn in-depth from the uniqueness of the single case itself.

Generalizing from the Single Case

A common approach to generalization and one most akin to a propositional form is cross-case generalization. In a collective or multi-site case study, each case is explored to see if issues that arise in one case also exist in other cases and what interconnecting themes there are between them. This kind of generalization has a degree of abstraction and potential for theorizing and is often welcomed by commissioners of research concerned that findings from the single case do not provide an adequate or “safe” basis for policy determination.

However, there are four additional ways to generalize from the single case, all of which draw more on tacit knowledge and recognition of context, although in different ways. In naturalistic generalization , first proposed by Stake (1978) , generalization is reached on the basis of recognition of similarities and differences to cases with which we are familiar. To enable such recognition, the case needs to feature rich description; people’s voices; and enough detail of time, place, and context to provide a vicarious experience to help readers discern what is similar and dissimilar to their own context ( Stake, 1978 ).

Situated generalization ( Simons, Kushner, Jones, & James, 2003 ) is close to the concept of naturalistic generalization in relying for its generality on retaining a connectedness with the context in which it first evolved. However, it has an extra dimension in a practice context. This notion of generalization was identified in an evaluation of a research project that engaged teachers in and with research. Here, in addition to the usual validity criteria to establish the warrant for the findings, the generalization was seen as dependable if trust existed between those who conducted the research (teachers, in this example) and those thinking about using it (other teachers). In other words, beyond the technical validity of the research, teachers considered using the findings in their own practice because they had confidence in those who generated them. This is a useful way to think about generalization if we wish research findings to improve professional practice.

The next two concepts of generalization— concept and process generalization —relate more to what you discover in making sense of the case. As you interpret and analyze, you begin to generate a theory of the case that makes sense of the whole. Concepts may be identified that make sense in the one case but have equal significance in other cases of a similar kind, even if the contexts are different.

It is the concept that generalizes, not the specific content or context. This may be similar to the process Donmoyer (2008) identifies of “intellectual generalization” (quoted by Butler-Kisber, 2010 , p. 15) to indicate the cognitive understanding one can gain from qualitative accounts even if settings are quite different.

The same is true for generalization of a process. It is possible to identify a significant process in one case (or several cases) that is transferable to other contexts, irrespective of the precise content and contexts of those other cases. An example here is the collaborative model for sustainable school self-evaluation I identified in researching school self-evaluation in a number of schools and countries ( Simons, 2002 ). Schools that successfully sustained school self-evaluation had an infrastructure that was collaborative at all stages of the evaluation process from design to conduct of the study, to analyzing the results and to reporting the findings. This ensured that the whole school was involved and that results were discussed and built into the ongoing development of school policies and practice. In other cases, different processes may be discovered that have applicability in a range of contexts. As with concept generalization, it is the process that generalizes not the substantive content or specific context.

Particularization

The forms of generalization discussed above are useful when we have to justify case study in a research or policy context. But the overarching justification for how we learn from case study is particularization —a rich portrayal of insights and understandings interpreted in the particular context. Several authors have made this point ( Stake, 1995 ; Flyvberg, 2006 ; Simons 2009 ). Stake puts it most sharply when he observes that “The real business of case study is particularization, not generalization” (p. 8), referring here to the main reason for studying the singular, which is to understand the uniqueness of the case itself.

My perspective (explored further in Simons, 1996 ; Simons, 2009 , p. 239; Simons & McCormack, 2007 ) is similar in that I believe the “real” strength of case study lies in the insights we gain from in-depth study of the particular. But I also argue for the universality of such insights—if we get it “right.” By which I mean that if we are able to capture and report the uniqueness, the essence, of the case in all its particularity and present this in a way we can all recognize, we will discover something of universal significance. This is something of a paradox. The more you learn in depth about the particularity of one person, situation, or context, the more likely you are to discover something universal. This process of reaching understanding has support both from the way in which many discoveries are made in science and in how we learn from artists, poets, and novelists, who reach us by communicating a recognizable truth about individuals, human relationships, and/or social contexts.

This concept of particularization is far from new, as the quotation from a preface to a book written in 1908 attests. Stephen Reynolds, the author of A Poor Man’s House , notes that the substance of the book was first recorded in a journal, kept for purposes of fiction, and in letters to one of his friends, but fiction proved an inappropriate medium. He felt that the life and the people were so much better than anything he could invent. The book therefore consists of the journal and letters drawn together to present a picture of a typical poor man’s house and life, much as we might draw together a range of data to present a case study. It is not the substance of the book that concerns us here but the methodological relevance to case study research. Reynolds notes that the conclusions expressed are tentative and possibly go beyond this man’s life, so he thought some explanation of the way he arrived at them was needed:

Educated people usually deal with the poor man’s life deductively; they reason from the general to the particular; and, starting with a theory, religious, philanthropic, political, or what not, they seek, and too easily find, among the millions of poor, specimens—very frequently abnormal—to illustrate their theories. With anything but human beings, that is an excellent method. Human beings, unfortunately, have individualities. They do what, theoretically, they ought not to do, and leave undone those things they ought to do. They are even said to possess souls—untrustworthy things beyond the reach of sociologists. The inductive method—reasoning from the particular to the general... should at least help to counterbalance the psychological superficiality of the deductive method. ( Reynolds, 1908 : preface) 1

Slightly overstated perhaps, but the point is well made. In our search for general laws, we not only lose sight of the uniqueness and humanity of individuals, but reduce them in the process, failing to present their experience in any “real” sense. What is astonishing about the quotation is that it was written over a century ago and yet many still argue today that you cannot generalize from the particular.

Going even further back, in 1798, Blake proclaimed that “To Generalize is to be an Idiot. To Particularize is the Alone Distinction of Merit.” In research, we may not wish to make such a strong distinction: these processes both have their uses in different kinds of research. But there is a major point here for the study of the particular that Wilson (2008) notes in commenting on Blake’s perception when he says: “Favouring the abstract over the concrete, one ‘sees all things only thro’ the narrow chinks of his cavern”’ (referring here to Blake’s The Marriage of Heaven and Hell [1793]; in Wilson, 2008 , p. 62). The danger Wilson is pointing to here is that abstraction relies heavily on what we know from our past understanding of things, and this may prevent us experiencing a concrete event directly or “apprehend[ing] a particular moment” ( Wilson, 2008 , p. 63).

Blake had a different mission, of course, than case researchers, and he was not himself free from abstractions, as Wilson points out, although he fought hard “to break through mental barriers to something unique and living” ( Wilson, 2008 , p. 65). It is this search for the “unique and living” and experiencing the “isness” of the particular that we should take from the Blake example to remind ourselves of the possibility of discovering something “new,” beyond our current understanding of the way things are.

Focusing on particularization does not diminish the usefulness of case study research for policy makers or practitioners. Grounded in recognizable experience, the potential is there to reach a range of audiences and to facilitate use of the findings. It may be more difficult for those who seek formal generalizations that seem to offer a safe basis for policy making to accept case study reports. However, particular stories often hold the key to why policies have or have not worked well in the past. It is not necessary to present long cases—a criticism frequently levelled—to demonstrate the story of the case. Such case stories can be most insightful for policy makers who, like many of us in everyday life, often draw inferences from a single instance or case, whatever the formal evidence presented. “I am reminded of the story of....”

The case for studying the particular to inform practice in professional contexts needs less persuasion because practitioners can recognize the content and context quite readily and make the inference to their own particular context ( Simons et al., 2003 ). In both sets of circumstances—policy and practice—it is more a question of whether the readers of our case research accept the validity of findings determined in this way, how they choose to learn, and our skill in telling the case study story.

Conclusion and Future Directions

In this chapter, I have presented an argument for case study research, making the case, in particular, for using qualitative methods to highlight what it is that qualitative case study research can bring to the study of social and educational programs. I outlined the various ways in which case study is commonly used before focusing directly on case study as a major mode of research inquiry, noting characteristics it shares with other qualitative methodologies, as well as itsdifference and the difficulties it is sometimes perceived to have. The chapter emphasizes the importance of thinking through what the case is, to be sure that the issues explored and the data generated do illuminate this case and not any other.

But there is still more to be done. In particular, I think we need to be more adventurous in how we craft and report the case. I suspect we may have been too cautious in the past in how we justified case study research, borrowing concepts from other disciplines and forms of educational research. More than 40 years on, it is time to take a greater risk—in demonstrating the intrinsic nature of case study and what it can offer to our understanding of human and social situations.

I have already drawn attention to the need to design the case, although this could be developed further to accentuate the uniqueness of the particular case. One way to do this is to feature individuals more in the design itself, not only to explore programs and policies through perspectives of key actors or groups and transactions between them, which to some extent happens already, but also to get them to characterize what makes the context unique. This is the reversal of many a design framework that starts with the logic of a program and takes forward the argument for personal evaluation ( Kushner, 2000 ), noted in the interlude on evaluation. Apart from this attention to design, there are three other issues I think we need to explore further: the warrant for creative methods in case study, more imaginative reporting; and how we learn from a study of the singular.

Warrant for More Creative Methods in Case Study Research

The promise that creative methods have for eliciting in-depth understanding and capturing the unusual, the idiosyncratic, the uniqueness of the case, was mentioned in the methods section. Yet, in case study research, particularly in program and policy contexts, we have few good examples of the use of artistic approaches for eliciting and interpreting data, although more, as acknowledged later, for presenting it. This may be because case study research is often conducted in academic or policy environments, where propositional ways of knowing are more valued.

Using creative and artistic forms in generating and interpreting case study data offers a form of evidence that acknowledges experiential understanding in illuminating the uniqueness of the case. The question is how to establish the warrant for this way of knowing and persuade others of its virtue. The answer is simple. By demonstrating the use of these methods in action, by arguing for a different form of validity that matches the intrinsic nature of the method, and, above all, by good examples.

Representing Findings to Engage Audiences in Learning

In evaluative and research policy contexts, where case study is often the main mode of inquiry or part of a broader study, case study reports often take a formal structure or sometimes, where the context is receptive, a portrayal or interpretative form. But, too often, the qualitative is an add-on to a story told by other means or reduced to issues in which the people who gave rise to the data are no longer seen. However, there are many ways to put them center stage.

Tell good stories and tell them well. Or, let key actors tell their own stories. Explore the different ways technology can help. Make video clips that demonstrate events in context, illustrate interactions between people, give voice to participants—show the reality of the program, in other words. Use graphics to summarize key issues and interactive, cartoon technology, as seen on some TED presentations, to summarize and visually show the complexity of the case. Video diaries were mentioned in the methods section: seeing individuals tell their tales directly is a powerful way of communicating, unhindered by “our” sense making. Tell photo stories. Let the photos convey the narrative, but make sure the structure of the narrative is evident to ensure coherence. These are just the beginnings. Those skilled in information technology could no doubt stretch our imagination further.

One problem and a further question concerns our audiences. Will they accept these modes of communication? Maybe not, in some contexts. However, there are three points I wish to leave you with. First, do not presume that they won’t. If people are fully present in the story and the complexity is not diminished, those reading, watching, or hearing about the case will get the message. If you are worried about how commissioners might respond, remember that they are no different from any other stakeholder or participant when it comes to how they learn from human experience. Witness the reference to Okri (1997) earlier about how we learn.

Second, when you detect that the context requires a more formal presentation of findings, respond according to expectation but also include elements of other forms of presentation. Nudge a little in the direction of creativity. Third, simply take a chance, that risk I spoke about earlier. Challenge the status quo. Find situations and contexts where you can fully represent the qualitative nature of the experience in the cases you study with creative forms of interpretation and representation. And let the audience decide.

Learning from a Study of the Singular

Finally, to return to the issue of “generalization” in case study that worries some audiences. I pointed out in the generalization section several ways in which it is possible to generalize from case study research, not in a formal propositional sense or from a case to a population, but by retaining a connection with the context in which the generalization first arose—that is, to realize in-depth understanding in context in different circumstances and situations. However, I also emphasized that, in many instances, it is particularization from which we learn. That is the point of the singular case study, and it is an art to perceive and craft the case in ways that we can.

Acknowledgments

Parts of this chapter build on ideas first explored in Simons, 2009 .

I am grateful to Bob Williams for pointing out the relevance of this quotation from Reynolds to remind us that “there is nothing new under the sun” and that we sometimes continue to engage endlessly in debates that have been well rehearsed before.

Adams, T. ( 2012 ) ‘ Olympics 2012: Team GB falters but London shines bright on opening day ’, Observer, 29.07.12.

Google Scholar

Google Preview

Bassey, M. ( 1999 ). Case study research in educational settings . Buckingham: Open University Press.

Bellah, R. N. , Madsen, R. , Sullivan, W. M. , Swidler, A. , & Tipton, S. M. ( 1985 ). Habits of the heart . London: Harper and Row.

Blake, W. (1798– 1809 ). Annotations to Sir Joshua Reynolds’s Discourses , pp. xvii–xcviii (c. 1798–1809) repr. In Complete Writings , ed.   Geoffrey Keynes   (1957). ‘Discourse II,’ annotations to Sir Joshua Reynolds, Discourses (c. 1808) .

Butler-Kisber, L. ( 2010 ). Qualitative inquiry: Thematic, narrative and arts-informed perspectives . London: Sage.

Cancienne, M. B. , & Snowber, C. N. ( 2003 ). Writing rhythm: Movement as method.   Qualitative Inquiry , 9 (2), 237–253.

Caulley, D. N. ( 2008 ). Making qualitative research reports less boring: The techniques of writing creative nonfiction.   Qualitative Inquiry , 14 (3) pp. 424–449.

Chadderton, C. , & Torrance, H. ( 2011 ). Case study. In B. Somekh & C. Lewin (Eds.), Theory and methods in social research . (2nd ed. pp 53–60). London and Thousand Oaks, CA: Sage.

Clandinin, D. J. & Connelly, F. M. ( 2000 ) Narrative inquiry: Experience and story in qualitative research. Ist edn. SanFrancisco: Jossey Bass Publishers.

Collier, J., Jr. ( 1967 ). Visual anthropology: Photography as a research method . New York: Holt, Reinhart, & Winston.

Denzin, N. K. & Lincoln, Y. S. (Eds.) ( 1994 ) Handbook of Qualitative Research , London and Thousand Oaks, CA:Sage

Donmoyer, R. ( 1990 ). Generalization and the single case study. In E. W. Eisner & A. Peshkin (Eds.), Qualitative inquiry in education (pp. 175–200). New York: Teachers College Press.

Donmoyer, R. ( 2008 ). Generalizability. In L. M. Givens (Ed.), The Sage encyclopedia of qualitative inquiry (vol. 2, pp. 371–372). Thousand Oaks, CA: Sage.

Duke, S. (2007). A narrative case study evaluation of the role of the Nurse Consultant in palliative care. PhD thesis, University of Southampton, England.

Elliott, J. (2008). Dance mirrors: Embodying, actualizing and operationalizing a dance experience in a healthcare context. PhD thesis, University of Ulster, Belfast.

Frank. A. ( 1997 ). Enacting illness stories: When, what, why. In H. L. Nelson (Ed.), Stories and their limits (pp. 31–49). London: Routledge.

Flyvberg, B. ( 2006 ). Five misunderstandings about case-study research.   Qualitative Inquiry , 12 (2), 219–245.

Gomm, R. , Hammersley, M. , & Foster, P. (Eds.). ( 2004 ). Case study method: Key issues, key texts . [First published in 2000]. London and Thousand Oaks, CA: Sage.

Greene, J. C. ( 2000 ). Understanding social programs through evaluation. In N. K. Denzin and Y. S. Lincoln (Eds.), The handbook of qualitative research (2nd ed., pp. 981–999). Thousand Oaks, CA: Sage.

Greene, J. C. ( 2007 ). Mixing methods in social inquiry . San Francisco: Jossey Bass.

Greenhalgh, T. , & Worrall, J. G. ( 1997 ). From EBM to CSM: The evolution of context-sensitive medicine.   Journal of Evaluation in Clinical Practice , 3 (2), 105–108.

Harrington, W. ( 2003 ). What journalism can offer ethnography.   Qualitative Inquiry . 9 (1), 90–114.

Heron, J. ( 1992 ). Feeling and personhood . Sage: London.

Heron, J. ( 1999 ). The complete facilitator’s handbook . London: Kogan Page.

House, E. R. ( 1980 ). Evaluating with validity . London, Beverly Hills, CA: Sage.

House, E. R. ( 1993 ). Professional evaluation: Social impact and political consequences . Newbury Park and London: Sage

Kushner, S. ( 2000 ). Personalizing evaluation . London and Thousand Oaks, CA: Sage.

Liamputtong, P. , & Rumbold, J. (Eds.). ( 2008 ). Knowing differently: Arts-based and collaborative research methods . New York: Nova Science Publishers.

MacDonald, B. ( 1976 ). Evaluation and the control of education. In D. Tawney (Ed.), Curriculum evaluation today: Trends and implications . Schools Council Research Studies (pp. 125– 136). London: Macmillan.

Matarasso, F. ( 2012 ). West Bromwich Operatic Society: fine art of musical theatre . West Bromwich, UK: Multistory.

Merriam, S. B. ( 1988 ). Case study research in education: A qualitative Approach . San Francisco: Jossey Bass.

Mullen, C. A. , & Finley, S. (Eds.). ( 2003 ). Arts-based approaches to qualitative inquiry [Special Issue].   Qualitative Inquiry , 9 (2), 165–329.

Okri, B. ( 1997 ). A way of being free . London: Phoenix.

Parlett, M. , & Hamilton, D. ( 1976 ). Evaluation as illumination: A new approach to the study of innovatory programmes. In G. Glass (Ed.), Evaluation studies review annual, I (pp. 140–157). [First published in 1972 as Occasional Paper 9, Centre for Research in the Educational Sciences, University of Edinburgh.] Beverly Hills: CA: Sage.

Platt, J. ( 2007 ). Case study. In W. Outhwaite & S. P. Turner (Eds.), The Sage handbook of social science methodology (pp. 100–118). London: Sage.

Prosser, J. ( 2000 ). The moral maze of image ethics. In H. Simons & R. Usher (Eds.), Situated ethics in educational research (pp. 116–132). London and New York: Routledge/Falmer.

Ragin, C. C. ( 1992 ). Cases of “What is a case?” In C. C. Ragin & H. S. Becker (Eds.), What is a case?: Exploring the foundations of social inquiry (pp. 1–17). Cambridge: Cambridge University Press.

Reynolds, S. S. (1908). A Poor Man’s House . The Project Guttenberg eBook, July 25, 2008 [eBook#26126]. Accessed February 26, 2013, http.//www.gutenberg.org

Richardson, L. ( 1994 ). Writing as a form of inquiry. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of Qualitative Research (pp. 516–529). London: Sage.

Richardson, L. ( 1997 ). Fields of play (Constructing an academic life) . New Brunswick, NJ: Rutgers University Press.

Rugang, L. (2006). Chinese culture in globalisation: A multi-modal case study on visual discourse. PhD thesis, University of Southampton, England.

Seale, C. ( 1999 ). The quality of qualitative research . London and Thousand Oaks, CA: Sage.

Sewell, K. ( 2011 ). Researching sensitive issues: A critical appraisal of “draw and write” as a data collection technique in eliciting children’s perceptions.   International Journal of Research Methods in Education 34(2), pp.175–191.

Shaw, I. , & Gould, N. ( 2001 ). Qualitative research in social work: Context and method . London: Sage.

Silverman, D. ( 2000 ). Doing qualitative research: A practical handbook . London and Thousand Oaks. CA: Sage.

Silverman, D. (Ed.). ( 2004 ). Qualitative research: Theory, methods and practice (2nd ed.). London and Thousand Oaks, CA: Sage.

Simons, H. ( 1971 ). Innovation and the case study of schools.   Cambridge Journal of Education , 3 , 118–123.

Simons, H. (Ed.). ( 1980 ). Towards a science of the singular: Essays about case study in educational research and evaluation . Occasional Papers No. 10. Norwich, UK: Centre for Applied Research, University of East Anglia.

Simons, H. ( 1987 ). Getting to know schools in a democracy: The politics and process of evaluation . Lewes, UK: Falmer Press.

Simons, H. ( 1996 ). The paradox of case study.   Cambridge Journal of Education , 26 (2), 225–240.

Simons, H. ( 2000 ). Damned if you do, damned if you don’t: ethical and political dilemmas in evaluation. In H. Simons & R. Usher (Eds.) Situated ethics in educational research (pp.39– 55) London and New York: Routledge/Falmer.

Simons, H. ( 2002 ). School self-evaluation in a democracy. In D. Nevo (Ed.), School-based evaluation: An international perspective . Advances in Program Evaluation. London: Sage.

Simons, H. ( 2006 ). Ethics and evaluation. In I. F. Shaw , J. C. Greene , & M. M. Mark (Eds.), The international handbook of evaluation (pp. 243–265). London and Thousand Oaks, CA Sage.

Simons, H. ( 2009 ). Case study research in practice . London: Sage.

Simons, H. (2010). Democratic evaluation: Theory and practice. Paper prepared for Virtual Evaluation Conference, University of the Witwatersrand, Johannesburg, South Africa, May, 2010.

Simons, H. , Kushner, S. , Jones, K. , & James, D. ( 2003 ). From evidence-based practice to practice-based evidence: The idea of situated generalization.   Research Papers in Education: Policy and Practice , 18 (4), 347–364.

Simons, H. , & McCormack, B. ( 2007 ). Integrating arts-based inquiry in evaluation methodology.   Qualitative Inquiry , 13 (2) 292–311.

Simons, H. , & Usher, R. (Eds.). ( 2000 ). Situated ethics in educational research . London and New York: Routledge/Falmer.

Smith, L. M. , & Pohland, P. A. ( 1974 ). Education, technology, and the rural highlands. In R. H. P. Kraft ., L. M. Smith ., P. A. Pohland ., C. J. Brauner , & C. Gjerde (Eds.), Four evaluation examples: Anthropological, economic, narrative and portrayal (pp. 5–54), AERA Monograph Series on Curriculum Evaluation 7. Chicago: Rand McNally.

Sparkes, A. ( 2002 ). Telling tales in sport and physical activity: A qualitative journey . Champaign, IL: Human Kinetics Press.

Sparkes, A. C. , & Douglas, K. ( 2007 ). Making the case for poetic representations: An example in action.   The Sport Psychologist , 21(2) , 170–190.

Spouse, J. ( 2000 ). Talking pictures: Investigating personal knowledge though illuminating artwork.   Nursing Times Research Journal , 5 (4), 253–261.

Stake, R. E. ( 1978 ) The case study method in social inquiry.   Educational Researcher , 7(2), 5–9.

Stake, R. E. ( 1995 ). The art of case study research . Thousand Oaks, CA and London: Sage.

Stake, R. E. ( 2010 ). Qualitative research: Studying how things work . New York & London: Guildford Press.

Stenhouse, L. ( 1978 ). Case study and case records: Towards a contemporary history of education.   British Educational Research Journal , 4 (2), 21–39.

Thomas, G. ( 2011 b)). A typology for the case study in social science following a review of definition, discourse and structure.   Qualitative Inquiry , 17 (6) , 511–521.

Thomas, G. ( 2011 a). How to do your case study: A guide for students and researchers . London: Sage.

Walker, R. ( 1993 ). Finding a silent voice for the researcher: Using photographs in evaluation and research. In M. Schratz (Ed.), Qualitative voices in educational research (pp. 72–92). Lewes, UK: Falmer Press.

Wilson, E. G. ( 2008 ). Against happiness . New York: Sarah Crichton Books.

Yin, R. K. ( 2004 ). Case study research: Design and methods . Thousand Oaks, CA and London: Sage.

Zucker, D. M. ( 2001 ). Using case study methodology in nursing research.   Qualitative Report , 6 (2) June.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Research Writing and Analysis

• NVivo Group and Study Sessions
• SPSS This link opens in a new window
• Statistical Analysis Group sessions
• Using Qualtrics
• Dissertation and Data Analysis Group Sessions
• Defense Schedule - Commons Calendar This link opens in a new window
• Research Process Flow Chart
• Research Alignment Chapter 1 This link opens in a new window
• Step 1: Seek Out Evidence
• Step 2: Explain
• Step 3: The Big Picture
• Step 4: Own It
• Step 5: Illustrate
• Annotated Bibliography
• Literature Review This link opens in a new window
• Systematic Reviews & Meta-Analyses
• How to Synthesize and Analyze
• Synthesis and Analysis Practice
• Synthesis and Analysis Group Sessions
• Problem Statement
• Purpose Statement
• Conceptual Framework
• Theoretical Framework
• Quantitative Research Questions
• Qualitative Research Questions
• Trustworthiness of Qualitative Data
• Analysis and Coding Example- Qualitative Data
• Thematic Data Analysis in Qualitative Design
• Dissertation to Journal Article This link opens in a new window
• International Journal of Online Graduate Education (IJOGE) This link opens in a new window
• Journal of Research in Innovative Teaching & Learning (JRIT&L) This link opens in a new window

Writing a Case Study

What is a case study?

A Case study is: 

• An in-depth research design that primarily uses a qualitative methodology but sometimes​​ includes quantitative methodology.
• Used to examine an identifiable problem confirmed through research.
• Used to investigate an individual, group of people, organization, or event.
• Used to mostly answer "how" and "why" questions.

What are the different types of case studies?

Note: These are the primary case studies. As you continue to research and learn

about case studies you will begin to find a robust list of different types. 

Who are your case study participants?

What is triangulation ? 

Validity and credibility are an essential part of the case study. Therefore, the researcher should include triangulation to ensure trustworthiness while accurately reflecting what the researcher seeks to investigate.

How to write a Case Study?

When developing a case study, there are different ways you could present the information, but remember to include the five parts for your case study.

Was this resource helpful?

• << Previous: Thematic Data Analysis in Qualitative Design
• Next: Journal Article Reporting Standards (JARS) >>
• Last Updated: May 16, 2024 8:25 AM
• URL: https://resources.nu.edu/researchtools

The Case Study as Research Method: A Practical Handbook

Qualitative Research in Accounting & Management

ISSN : 1176-6093

Article publication date: 21 June 2011

Scapens, R.W. (2011), "The Case Study as Research Method: A Practical Handbook", Qualitative Research in Accounting & Management , Vol. 8 No. 2, pp. 201-204. https://doi.org/10.1108/11766091111137582

Emerald Group Publishing Limited

Copyright © 2011, Emerald Group Publishing Limited

This book aims to provide case‐study researchers with a step‐by‐step practical guide to “help them conduct the study with the required degree of rigour” (p. xi).

It seeks to “demonstrate that the case study is indeed a scientific method” (p. 104) and to show “the usefulness of the case method as one tool in the researcher's methodological arsenal” (p. 105). The individual chapters cover the various stages in conducting case‐study research, and each chapter sets out a number of practical steps which have to be taken by the researcher. The following are the eight stages/chapters and, in brackets, the number of steps in each stages:

Assessing appropriateness and usefulness (4).

Ensuring accuracy of results (21).

Preparation (6).

Selecting cases (4).

Collecting data (7).

Analyzing data (4).

Interpreting data (3).

Reporting results (4).

It is particularly noticeable that ensuring accuracy of results has by far the largest number of number of steps – 21 steps compared to seven or fewer steps in the other stages. This reflects Gagnon's concern to demonstrate the scientific rigour of case‐study research. In the forward, he explains that the book draws on his experience in conducting his own PhD research, which was closely supervised by three professors, one of whom was inclined towards quantitative research. Consequently, his research was underpinned by the principles and philosophy of quantitative research. This is clearly reflected in the approach taken in this book, which seeks to show that case‐study research is just as rigorous and scientific as quantitative research, and it can produce an objective and accurate representation of the observed reality.

There is no discussion of the methodological issues relating to the use of case‐study research methods. This is acknowledged in the forward, although Gagnon refers to them as philosophical or epistemological issues (p. xii), as he tends to use the terms methodology and method interchangeably – as is common in quantitative research. Although he starts (step 1.1) by trying to distance case and other qualitative research from the work of positivists, arguing that society is socially constructed, he nevertheless sees social reality as objective and independent of the researcher. So for Gagnon, the aim of case research is to accurately reflect that reality. At various points in the book the notion of interpretation is used – evidence is interpreted and the (objective) case findings have to be interpreted.

So although there is a distancing from positivist research (p. 1), the approach taken in this book retains an objective view of the social reality which is being researched; a view which is rather different to the subjective view of reality taken by many interpretive case researchers. This distinction between an objective and a subjective view of the social reality being researched – and especially its use in contrasting positivist and interpretive research – has its origins the taxonomy of Burrell and Morgan (1979) . Although there have been various developments in the so‐called “objective‐subjective debate”, and recently some discussion in relation to management accounting research ( Kakkuri‐Knuuttila et al. , 2008 ; Ahrens, 2008 ), this debate is not mentioned in the book. Nevertheless, it is clear that Gagnon is firmly in the objective camp. In a recent paper, Johnson et al. (2006, p. 138) provide a more contemporary classification of the different types of qualitative research. In their terms, the approach taken in this book could be described as neo‐empiricist – an approach which they characterise as “qualitative positivists”.

The approach taken in this handbook leaves case studies open to the criticisms that they are a small sample, and consequently difficult to generalise, and to arguments that case studies are most appropriate for exploratory research which can subsequently be generalised though quantitative research. Gagnon explains that this was the approach he used after completing his thesis (p. xi). The handbook only seems to recognise two types of case studies, namely exploratory and raw empirical case studies – the latter being used where “the researcher is interested in a subject without having formed any preconceived ideas about it” (p. 15) – which has echoes of Glaser and Strauss (1967) . However, limiting case studies to these two types ignores other potential types; in particular, explanatory case studies which are where interpretive case‐study research can make important contributions ( Ryan et al. , 2002 ).

This limited approach to case studies comes through in the practical steps which are recommended in the handbook, and especially in the discussion of reliability and validity. The suggested steps seem to be designed to keep very close to the notions of reliability and validity used in quantitative research. There is no mention of the recent discussion of “validity” in interpretive accounting research, which emphasises the importance of authenticity and credibility and their implications for writing up qualitative and case‐study research ( Lukka and Modell, 2010 ). Although the final stage of Gagnon's handbook makes some very general comments about reporting the results, it does not mention, for example, Baxter and Chua's (2008) paper in QRAM which discusses the importance of demonstrating authenticity, credibility and transferability in writing qualitative research.

Despite Gagnon's emphasis on traditional notions of reliability and validity the handbook provides some useful practical advice for all case‐study researchers. For example, case‐study research needs a very good research design; case‐study researchers must work hard to gain access to and acceptance in the research settings; a clear strategy is needed for data collection; the case researcher should create field notes (in a field notebook, or otherwise) to record all the thoughts, ideas, observations, etc. that would not otherwise be collected; and the vast amount of data that case‐study research can generate needs to be carefully managed. Furthermore, because of what Gagnon calls the “risk of mortality” (p. 54) (i.e. the risk that access to a research site may be lost – for instance, if the organisation goes bankrupt) it is crucial for some additional site(s) to be selected at the outset to ensure that the planned research can be completed. This is what I call “insurance cases” when talking to my own PhD students. Interestingly, Gagnon recognises the ethical issues involved in doing case studies – something which is not always mentioned by the more objectivist type of case‐study researchers. He emphasises that it is crucial to honour confidentiality agreements, to ensure data are stored securely and that commitments are met and promises kept.

There is an interesting discussion of the advantages and disadvantages of using computer methods in analysing data (in stage 6). However, the discussion of coding appears to be heavily influenced by grounded theory, and is clearly concerned with producing an accurate reflection of an objective reality. In addition, Gagnon's depiction of case analysis is overly focussed on content analysis – possibly because it is a quantitative type of technique. There is no reference to the other approaches available to qualitative researchers. For example, there is no mention of the various visualisation techniques set out in Miles and Huberman (1994) .

To summarise, Gagnon's book is particularly useful for case‐study researchers who see the reality they are researching as objective and researcher independent. However, this is a sub‐set of case‐study researchers. Although some of the practical guidance offered is relevant for other types of case‐study researchers, those who see multiple realities in the social actors and/or recognise the subjectivity of the research process might have difficulty with some of the steps in this handbook. Gagnon's aim to show that the case study is a scientific method, gives the handbook a focus on traditional (quantitatively inspired) notions rigour and validity, and a tendency to ignore (or at least marginalise) other types of case study research. For example, the focus on exploratory cases, which need to be supplemented by broad based quantitative research, overlooks the real potential of case study research which lies in explanatory cases. Furthermore, Gagnon is rather worried about participant research, as the researcher may play a role which is “not consistent with scientific method” (p. 42), and which may introduce researcher bias and thereby damage “the impartiality of the study” (p. 53). Leaving aside the philosophical question about whether any social science research, including quantitative research, can be impartial, this stance could severely limit the potential of case‐study research and it would rule out both the early work on the sociology of mass production and the recent calls for interventionist research. Clearly, there could be a problem where a researcher is trying to sell consulting services, but there is a long tradition of social researchers working within organisations that they are studying. Furthermore, if interpretive research is to be relevant for practice, researchers may have to work with organisations to introduce new ideas and new ways of analysing problems. Gagnon would seem to want to avoid all such research – as it would not be “impartial”.

Consequently, although there is some good practical advice for case study researchers in this handbook, some of the recommendations have to be treated cautiously, as it is a book which sees case‐study research in a very specific way. As mentioned earlier, in the Forward Gagnon explicitly recognises that the book does not take a position on the methodological debates surrounding the use of case studies as a research method, and he says that “The reader should therefore use and judge this handbook with these considerations in mind” (p. xii). This is very good advice – caveat emptor .

Ahrens , T. ( 2008 ), “ A comment on Marja‐Liisa Kakkuri‐Knuuttila ”, Accounting, Organizations and Society , Vol. 33 Nos 2/3 , pp. 291 ‐ 7 , Kari Lukka and Jaakko Kuorikoski.

Baxter , J. and Chua , W.F. ( 2008 ), “ The field researcher as author‐writer ”, Qualitative Research in Accounting & Management , Vol. 5 No. 2 , pp. 101 ‐ 21 .

Burrell , G. and Morgan , G. ( 1979 ), Sociological Paradigms and Organizational Analysis , Heinneman , London .

Glaser , B.G. and Strauss , A.L. ( 1967 ), The Discovery of Grounded Theory: Strategies for Qualitative Research , Aldine , New York, NY .

Johnson , P. , Buehring , A. , Cassell , C. and Symon , G. ( 2006 ), “ Evaluating qualitative management research: towards a contingent critieriology ”, International Journal of Management Reviews , Vol. 8 No. 3 , pp. 131 ‐ 56 .

Kakkuri‐Knuuttila , M.‐L. , Lukka , K. and Kuorikoski , J. ( 2008 ), “ Straddling between paradigms: a naturalistic philosophical case study on interpretive research in management accounting ”, Accounting, Organizations and Society , Vol. 33 Nos 2/3 , pp. 267 ‐ 91 .

Lukka , K. and Modell , S. ( 2010 ), “ Validation in interpretive management accounting research ”, Accounting, Organizations and Society , Vol. 35 , pp. 462 ‐ 77 .

Miles , M.B. and Huberman , A.M. ( 1994 ), Qualitative Data Analysis: A Source Book of New Methods , 2nd ed. , Sage , London .

Ryan , R.J. , Scapens , R.W. and Theobald , M. ( 2002 ), Research Methods and Methodology in Finance and Accounting , 2nd ed. , Thomson Learning , London .

Related articles

We’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Case Study | Definition, Examples & Methods

Case Study | Definition, Examples & Methods

Published on 5 May 2022 by Shona McCombes . Revised on 30 January 2023.

A case study is a detailed study of a specific subject, such as a person, group, place, event, organisation, or phenomenon. Case studies are commonly used in social, educational, clinical, and business research.

A case study research design usually involves qualitative methods , but quantitative methods are sometimes also used. Case studies are good for describing , comparing, evaluating, and understanding different aspects of a research problem .

Table of contents

When to do a case study, step 1: select a case, step 2: build a theoretical framework, step 3: collect your data, step 4: describe and analyse the case.

A case study is an appropriate research design when you want to gain concrete, contextual, in-depth knowledge about a specific real-world subject. It allows you to explore the key characteristics, meanings, and implications of the case.

Case studies are often a good choice in a thesis or dissertation . They keep your project focused and manageable when you don’t have the time or resources to do large-scale research.

You might use just one complex case study where you explore a single subject in depth, or conduct multiple case studies to compare and illuminate different aspects of your research problem.

Prevent plagiarism, run a free check.

Once you have developed your problem statement and research questions , you should be ready to choose the specific case that you want to focus on. A good case study should have the potential to:

• Provide new or unexpected insights into the subject
• Challenge or complicate existing assumptions and theories
• Propose practical courses of action to resolve a problem
• Open up new directions for future research

Unlike quantitative or experimental research, a strong case study does not require a random or representative sample. In fact, case studies often deliberately focus on unusual, neglected, or outlying cases which may shed new light on the research problem.

If you find yourself aiming to simultaneously investigate and solve an issue, consider conducting action research . As its name suggests, action research conducts research and takes action at the same time, and is highly iterative and flexible. 

However, you can also choose a more common or representative case to exemplify a particular category, experience, or phenomenon.

While case studies focus more on concrete details than general theories, they should usually have some connection with theory in the field. This way the case study is not just an isolated description, but is integrated into existing knowledge about the topic. It might aim to:

• Exemplify a theory by showing how it explains the case under investigation
• Expand on a theory by uncovering new concepts and ideas that need to be incorporated
• Challenge a theory by exploring an outlier case that doesn’t fit with established assumptions

To ensure that your analysis of the case has a solid academic grounding, you should conduct a literature review of sources related to the topic and develop a theoretical framework . This means identifying key concepts and theories to guide your analysis and interpretation.

There are many different research methods you can use to collect data on your subject. Case studies tend to focus on qualitative data using methods such as interviews, observations, and analysis of primary and secondary sources (e.g., newspaper articles, photographs, official records). Sometimes a case study will also collect quantitative data .

The aim is to gain as thorough an understanding as possible of the case and its context.

In writing up the case study, you need to bring together all the relevant aspects to give as complete a picture as possible of the subject.

How you report your findings depends on the type of research you are doing. Some case studies are structured like a standard scientific paper or thesis, with separate sections or chapters for the methods , results , and discussion .

Others are written in a more narrative style, aiming to explore the case from various angles and analyse its meanings and implications (for example, by using textual analysis or discourse analysis ).

In all cases, though, make sure to give contextual details about the case, connect it back to the literature and theory, and discuss how it fits into wider patterns or debates.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2023, January 30). Case Study | Definition, Examples & Methods. Scribbr. Retrieved 14 May 2024, from https://www.scribbr.co.uk/research-methods/case-studies/

Is this article helpful?

Shona McCombes

Other students also liked, correlational research | guide, design & examples, a quick guide to experimental design | 5 steps & examples, descriptive research design | definition, methods & examples.

• First Online: 27 October 2022

Cite this chapter

• R. M. Channaveer 4 &
• Rajendra Baikady 5  

2557 Accesses

1 Citations

This chapter reviews the strengths and limitations of case study as a research method in social sciences. It provides an account of an evidence base to justify why a case study is best suitable for some research questions and why not for some other research questions. Case study designing around the research context, defining the structure and modality, conducting the study, collecting the data through triangulation mode, analysing the data, and interpreting the data and theory building at the end give a holistic view of it. In addition, the chapter also focuses on the types of case study and when and where to use case study as a research method in social science research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Ang, C. S., Lee, K. F., & Dipolog-Ubanan, G. F. (2019). Determinants of first-year student identity and satisfaction in higher education: A quantitative case study. SAGE Open, 9 (2), 215824401984668. https://doi.org/10.1177/2158244019846689

Baxter, P., & Jack, S. (2015). Qualitative case study methodology: Study design and implementation for novice researchers. The Qualitative Report . Published. https://doi.org/10.46743/2160-3715/2008.1573

Bhatta, T. P. (2018). Case study research, philosophical position and theory building: A methodological discussion. Dhaulagiri Journal of Sociology and Anthropology, 12 , 72–79. https://doi.org/10.3126/dsaj.v12i0.22182

Article   Google Scholar  

Bromley, P. D. (1990). Academic contributions to psychological counselling. A philosophy of science for the study of individual cases. Counselling Psychology Quarterly , 3 (3), 299–307.

Google Scholar  

Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A., & Sheikh, A. (2011). The case study approach. BMC Medical Research Methodology, 11 (1), 1–9.

Grässel, E., & Schirmer, B. (2006). The use of volunteers to support family carers of dementia patients: Results of a prospective longitudinal study investigating expectations towards and experience with training and professional support. Zeitschrift Fur Gerontologie Und Geriatrie, 39 (3), 217–226.

Greenwood, D., & Lowenthal, D. (2005). Case study as a means of researching social work and improving practitioner education. Journal of Social Work Practice, 19 (2), 181–193. https://doi.org/10.1080/02650530500144782

Gülseçen, S., & Kubat, A. (2006). Teaching ICT to teacher candidates using PBL: A qualitative and quantitative evaluation. Journal of Educational Technology & Society, 9 (2), 96–106.

Gomm, R., Hammersley, M., & Foster, P. (2000). Case study and generalization. Case study method , 98–115.

Hamera, J., Denzin, N. K., & Lincoln, Y. S. (2011). Performance ethnography . SAGE.

Hayes, N. (2000). Doing psychological research (p. 133). Open University Press.

Harrison, H., Birks, M., Franklin, R., & Mills, J. (2017). Case study research: Foundations and methodological orientations. In Forum qualitative sozialforschung/forum: Qualitative social research (Vol. 18, No. 1).

Iwakabe, S., & Gazzola, N. (2009). From single-case studies to practice-based knowledge: Aggregating and synthesizing case studies. Psychotherapy Research, 19 (4–5), 601–611. https://doi.org/10.1080/10503300802688494

Johnson, M. P. (2006). Decision models for the location of community corrections centers. Environment and Planning b: Planning and Design, 33 (3), 393–412. https://doi.org/10.1068/b3125

Kaarbo, J., & Beasley, R. K. (1999). A practical guide to the comparative case study method in political psychology. Political Psychology, 20 (2), 369–391. https://doi.org/10.1111/0162-895x.00149

Lovell, G. I. (2006). Justice excused: The deployment of law in everyday political encounters. Law Society Review, 40 (2), 283–324. https://doi.org/10.1111/j.1540-5893.2006.00265.x

McDonough, S., & McDonough, S. (1997). Research methods as part of English language teacher education. English Language Teacher Education and Development, 3 (1), 84–96.

Meredith, J. (1998). Building operations management theory through case and field research. Journal of Operations Management, 16 (4), 441–454. https://doi.org/10.1016/s0272-6963(98)00023-0

Mills, A. J., Durepos, G., & Wiebe, E. (Eds.). (2009). Encyclopedia of case study research . Sage Publications.

Ochieng, P. A. (2009). An analysis of the strengths and limitation of qualitative and quantitative research paradigms. Problems of Education in the 21st Century , 13 , 13.

Page, E. B., Webb, E. J., Campell, D. T., Schwart, R. D., & Sechrest, L. (1966). Unobtrusive measures: Nonreactive research in the social sciences. American Educational Research Journal, 3 (4), 317. https://doi.org/10.2307/1162043

Rashid, Y., Rashid, A., Warraich, M. A., Sabir, S. S., & Waseem, A. (2019). Case study method: A step-by-step guide for business researchers. International Journal of Qualitative Methods, 18 , 160940691986242. https://doi.org/10.1177/1609406919862424

Ridder, H. G. (2017). The theory contribution of case study research designs. Business Research, 10 (2), 281–305. https://doi.org/10.1007/s40685-017-0045-z

Sadeghi Moghadam, M. R., Ghasemnia Arabi, N., & Khoshsima, G. (2021). A Review of case study method in operations management research. International Journal of Qualitative Methods, 20 , 160940692110100. https://doi.org/10.1177/16094069211010088

Sommer, B. B., & Sommer, R. (1997). A practical guide to behavioral research: Tools and techniques . Oxford University Press.

Stake, R. E. (2010). Qualitative research: Studying how things work .

Stake, R. E. (1995). The Art of Case Study Research . Sage Publications.

Stoecker, R. (1991). Evaluating and rethinking the case study. The Sociological Review, 39 (1), 88–112.

Suryani, A. (2013). Comparing case study and ethnography as qualitative research approaches .

Taylor, S., & Berridge, V. (2006). Medicinal plants and malaria: An historical case study of research at the London School of Hygiene and Tropical Medicine in the twentieth century. Transactions of the Royal Society of Tropical Medicine and Hygiene, 100 (8), 707–714. https://doi.org/10.1016/j.trstmh.2005.11.017

Tellis, W. (1997). Introduction to case study. The Qualitative Report . Published. https://doi.org/10.46743/2160-3715/1997.2024

Towne, L., & Shavelson, R. J. (2002). Scientific research in education . National Academy Press Publications Sales Office.

Widdowson, M. D. J. (2011). Case study research methodology. International Journal of Transactional Analysis Research, 2 (1), 25–34.

Yin, R. K. (2004). The case study anthology . Sage.

Yin, R. K. (2003). Design and methods. Case Study Research , 3 (9.2).

Yin, R. K. (1994). Case study research: Design and methods (2nd ed.). Sage Publishing.

Yin, R. (1984). Case study research: Design and methods . Sage Publications Beverly Hills.

Yin, R. (1993). Applications of case study research . Sage Publishing.

Zainal, Z. (2003). An investigation into the effects of discipline-specific knowledge, proficiency and genre on reading comprehension and strategies of Malaysia ESP Students. Unpublished Ph. D. Thesis. University of Reading , 1 (1).

Zeisel, J. (1984). Inquiry by design: Tools for environment-behaviour research (No. 5). CUP archive.

Download references

Author information

Authors and affiliations.

Department of Social Work, Central University of Karnataka, Kadaganchi, India

R. M. Channaveer

Department of Social Work, University of Johannesburg, Johannesburg, South Africa

Rajendra Baikady

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to R. M. Channaveer .

Editor information

Editors and affiliations.

Centre for Family and Child Studies, Research Institute of Humanities and Social Sciences, University of Sharjah, Sharjah, United Arab Emirates

M. Rezaul Islam

Department of Development Studies, University of Dhaka, Dhaka, Bangladesh

Niaz Ahmed Khan

Department of Social Work, School of Humanities, University of Johannesburg, Johannesburg, South Africa

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Channaveer, R.M., Baikady, R. (2022). Case Study. In: Islam, M.R., Khan, N.A., Baikady, R. (eds) Principles of Social Research Methodology. Springer, Singapore. https://doi.org/10.1007/978-981-19-5441-2_21

Download citation

DOI : https://doi.org/10.1007/978-981-19-5441-2_21

Published : 27 October 2022

Publisher Name : Springer, Singapore

Print ISBN : 978-981-19-5219-7

Online ISBN : 978-981-19-5441-2

eBook Packages : Social Sciences

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Open access
• Published: 17 May 2024

Weighted metrics are required when evaluating the performance of prediction models in nested case–control studies

• Barbara Rentroia-Pacheco 1 ,
• Domenico Bellomo 2 ,
• Inge M. M. Lakeman 3 , 4 ,
• Marlies Wakkee 1 ,
• Loes M. Hollestein 1 , 5   na1 &
• David van Klaveren 6   na1  

BMC Medical Research Methodology volume  24 , Article number:  115 ( 2024 ) Cite this article

Metrics details

Nested case–control (NCC) designs are efficient for developing and validating prediction models that use expensive or difficult-to-obtain predictors, especially when the outcome is rare. Previous research has focused on how to develop prediction models in this sampling design, but little attention has been given to model validation in this context. We therefore aimed to systematically characterize the key elements for the correct evaluation of the performance of prediction models in NCC data.

We proposed how to correctly evaluate prediction models in NCC data, by adjusting performance metrics with sampling weights to account for the NCC sampling. We included in this study the C-index, threshold-based metrics, Observed-to-expected events ratio (O/E ratio), calibration slope, and decision curve analysis. We illustrated the proposed metrics with a validation of the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA version 5) in data from the population-based Rotterdam study. We compared the metrics obtained in the full cohort with those obtained in NCC datasets sampled from the Rotterdam study, with and without a matched design.

Performance metrics without weight adjustment were biased: the unweighted C-index in NCC datasets was 0.61 (0.58–0.63) for the unmatched design, while the C-index in the full cohort and the weighted C-index in the NCC datasets were similar: 0.65 (0.62–0.69) and 0.65 (0.61–0.69), respectively. The unweighted O/E ratio was 18.38 (17.67–19.06) in the NCC datasets, while it was 1.69 (1.42–1.93) in the full cohort and its weighted version in the NCC datasets was 1.68 (1.53–1.84). Similarly, weighted adjustments of threshold-based metrics and net benefit for decision curves were unbiased estimates of the corresponding metrics in the full cohort, while the corresponding unweighted metrics were biased. In the matched design, the bias of the unweighted metrics was larger, but it could also be compensated by the weight adjustment.

Conclusions

Nested case–control studies are an efficient solution for evaluating the performance of prediction models that use expensive or difficult-to-obtain biomarkers, especially when the outcome is rare, but the performance metrics need to be adjusted to the sampling procedure.

Peer Review reports

Risk prediction models are becoming increasingly popular in the medical community to predict clinical outcomes and can be used to provide more personalized decisions to patients. Population-based longitudinal cohorts, with information on thousands of individuals, are now widespread. These cohorts enable the identification of individuals with a disease and their biological samples at the population level, as well as the development of (risk) prediction models [ 1 , 2 ]. While population-based cohorts are the preferred study design for building and validating such models [ 3 , 4 ], they are expensive, particularly when the collected data go beyond routinely available variables (e.g. demographic or basic clinical characteristics), and include additional information (e.g. measurements of expensive or difficult-to-obtain biomarkers). In addition, collecting this information for all subjects might not always be feasible. Using the full cohort is, therefore, functional for unbiased sample identification, but not efficient for model development and validation, particularly for rare outcomes. These scenarios require more efficient study designs, such as nested case–control (NCC) and case-cohort designs [ 3 , 4 ].

In NCC studies, a subsample of a fully enumerated source population is identified, containing all patients who experience the outcome of interest during the study follow-up period (cases), together with a sample of patients who do not experience the outcome during the time-at-risk (controls) [ 4 , 5 ]. Often, controls are matched to cases to reduce potential confounding [ 5 ]. Follow-up and matching variables must be available in the full cohort to identify the controls, whereas other variables of interest are only collected for the case–control set. This greatly reduces the amount of data collection needed (and the associated costs), while still providing accurate estimates of the effects of risk factors and performance estimates [ 4 , 6 ]. In contrast to the traditional case–control design, where cases and controls are sampled from a population of unknown size, in NCC studies, cases and controls are sampled from a well-defined population of known size (i.e. hence the designation “nested”) [ 3 ]. This makes them suitable to estimate hazard and odds ratios of the full cohort and to build absolute risk models [ 3 , 5 ], without the need for additional data sources. However, appropriate methodology must be employed to accommodate the under-sampling of controls and potentially any matching [ 5 , 7 , 8 ].

Clear recommendations on how to develop prediction models in NCC designs can be found in the literature [ 9 , 10 ], as well as some examples of risk prediction models for rare outcomes developed in NCC data [ 8 , 11 , 12 , 13 ]. However, we have noticed that there is a lack of clear guidance on how to properly validate prediction models in this study design. This has led to incorrect performance estimates of these models reported in several works in the literature [ 14 , 15 ], and to the misconception that NCC studies are not suited to validate prediction models [ 16 ]. Clear guidance on this matter would therefore be useful for clinical and methodological researchers evaluating the performance of prediction models in NCC data.

We aim to present the key elements for correct evaluation of the performance of prediction models in NCC datasets. We propose how relevant model performance metrics should be adjusted to compensate for 1) the overrepresentation of cases in the NCC dataset and 2) the fact that the controls in the NCC dataset are no longer representative of all controls in the source population. We then illustrate the importance of using adjusted performance metrics with a real world example, where we validate the well-known BOADICEA [ 17 ] breast cancer risk prediction model in NCC datasets, and compare the obtained performance metrics to those that were reported in the original full cohort study [ 18 ].

Validating risk prediction models in nested case–control data

In this study, we focus on the scenario where a prediction model has already been developed, in a population-based cohort or in another design, and we aim to evaluate its performance in an NCC design. Table 1 describes how NCC datasets are obtained using incidence density sampling, and Fig.  1 illustrates two completely worked out examples for an unmatched (A) and matched design (B) respectively. Of note, under this sampling method, the same subject can be selected multiple times: cases can be selected as controls of other cases with shorter event times; controls can be selected in risk sets of different cases. However, for model validation in NCC datasets, including the same subject multiple times can lead to biased estimates of performance metrics [ 4 ]. Therefore, only one record should be kept for each subject: the case record for subjects selected as cases, and a randomly selected control record for controls [ 4 ].

Derivation of a nested case–control dataset from a full cohort, using incidence density sampling. In A , no matching variable is used. In B , sex is used to match controls to cases. Step 1. Required information for all subjects in the source population (full cohort) is extracted: outcome of interest, follow up time and, if needed, additional matching variables. Time-to-event plot represents timelines of subjects in the full cohort (each row represents one subject). Cases (in red) correspond to subjects who experience the outcome of interest. Controls (in yellow) correspond to subjects who do not experience the outcome of interest. Crosses represent last time of follow-up for controls. Sex (male (M) or female (F)) is indicated in purple. Step 2: Control sampling using incidence density sampling: for each case, the subjects who have not experienced any event at the time of the event for that case (subjects-at-risk) are identified, and one or more subjects-at-risk are randomly sampled. In this example, the sampling ratio is 1:1, therefore only 1 control is sampled per case (indicated with a light blue circle in A and a purple circle in B ). The number of subjects-at-risk in step 2 in the matched scenario ( B ) is typically smaller than in the scenario without matching ( A ). Note that, under incidence density sampling, some cases can be sampled as controls if they are part of the subjects-at-risk for another case before they experience the event of interest. Subjects can also be sampled more than once as controls. Other sampling methods exist, but incidence density sampling is the method that leads to more unbiased estimates in NCC datasets

Challenges in validating prediction models in NCC data

Validating prediction models in NCC datasets cannot be performed in the same way as in full cohorts for two reasons. First, the sampling procedure artificially increases the proportion of cases (Fig.  2 A). The difference between the original proportion and the one in the NCC dataset will depend on the chosen sampling ratio (number of controls sampled per case). Second, the controls in the NCC dataset are no longer representative of all controls: even if no matching is performed, subjects with longer survival time are more likely to be included in the NCC dataset as controls. This can distort the distributions of the risk factors in the NCC dataset (Fig.  2 B). Failing to account for this distortion will result in overestimation of the absolute risk, since controls are underrepresented in the NCC dataset compared to the full cohort, and in the inaccurate estimation of both risk factor effects (e.g., odds ratios) and performance metrics.

Characteristics of nested case–control (NCC) datasets that affect model validation ( A ) Comparison of proportions of cases (in red) and of controls (in yellow) in the full cohort and the NCC data. Outcome prevalence is distorted in the NCC dataset, compared to the full cohort. The magnitude of the distortion depends on the sampling ratio (number of controls sampled for each case). ( B ) NCC sampling can distort risk factor distribution in sampled controls. For example, we might be studying a clinical outcome that is more prevalent in females (in pink) than males (in blue), with 75% of events occurring in females (3 out of 4 events). However, in subjects without events sex is evenly distributed (48 females + 48 males). In the study, males are more likely to leave the study earlier than females, leading to shorter follow up times for males. This results in females being more likely to be sampled as controls: the proportion of females in the NCC controls is 75% instead of 50% observed in the full cohort. Reweighting the NCC data can reconstruct the original distribution of sex in the sampled controls (50% males, 50% females) and allow the unbiased estimation of odds ratios and performance metrics

However, if subjects are weighted based on their sampling probability, NCC datasets can be used to develop and validate prediction models. The weighting process enables the recovery of the original ratio of cases to controls and the original risk factor distributions, which consequently enables the development of models that predict absolute risk [ 5 , 10 ]. Additionally, in the validation of a model in an NCC dataset, the same weighting process can be applied to adjust the performance metrics, making it possible to evaluate model performance accurately, without applying the model to all subjects of the corresponding full cohort.

Assigning weights to subjects in NCC data

The weight of subject \(k\) \({(w}_{k})\) is computed as the inverse of the probability of subject \(k\) being included in the NCC dataset, and accounts for the sampling design, namely any matching performed, as well as the case–control sampling ratio. For a NCC dataset with \(K\) subjects ( \(J\) cases and \(I\) controls), the sampling probability of case \(j\) \({(p}_{j})\) is straightforward to compute. However, the sampling probabilities of controls, each denoted as \({p}_{i}\) , take a more complex form, as several factors can influence their selection.

Sampling probabilities of cases ( \({{\varvec{p}}}_{{\varvec{j}}}\) )

In a typical NCC study, all cases present in the full cohort are included in the NCC dataset; therefore, all cases have a sampling probability \({p}_{j}\) of 1. However, extracting model variables for all cases might not be possible due to time or cost constraints, leading to more atypical NCC studies, where only a portion of the cases are included. In this scenario, the sampling probability corresponds to the proportion of cases identified in the full cohort that are included in the NCC dataset [ 19 ].

Sampling probabilities of controls ( \({{\varvec{p}}}_{{\varvec{i}}}\) )

There are different ways of estimating the probability of sampling subject \(k\) as a control [ 20 ]. We focus on the Kaplan–Meier and the logistic/generalized additive model estimators, for their ease of implementation and ability to deal with matched NCC designs.

The sampling probability of control \(i\) ( \({p}_{i}\) ) estimated with the Kaplan–Meier estimate [ 20 ] uses the property that under the incidence density sampling scheme, the probabilities of a control being eligible for sampling for different cases are independent across the (matched) risk sets [ 21 ]. It is defined as follows:

where \(m\) is the number of controls sampled per case, \(I\) is an indicator variable (with value 1 if control \(i\) is eligible for case \(j\) , and 0 otherwise), \({t}_{j}\) is the event time of case \(j\) , and \({n}_{j}\left({t}_{j}\right)\) is the number of subjects at risk at time \({t}_{j}\) , which could be matched to case \(j\) .

Sampling probabilities can also be estimated using a model-based approach. For example, a logistic regression model can be fitted to predict the probability of a control being sampled, using its censoring time and additional matching variables as predictors. This logistic regression is fitted to the full cohort, excluding all cases. The sampling probability \({p}_{i}\) then corresponds to the output probability of this logistic regression applied to control \(i\) , and the inverse of such sampling probabilities are denominated generalized linear model (GLM) weights. Sampling probabilities can also be estimated based on a generalized additive model (GAM), leading to GAM weights.

These weighting schemes are described in more detail in Støer and Samuelsen [ 20 ]. Of note, other weighing methods exist, such as the Chen weights [ 20 ]. Despite their differences, there is little variation in model estimates and standard errors using different weighting schemes during model development [ 10 , 22 ]. All of these methods for weight computation can be implemented with the multipleNCC R package [ 20 ].

Inverse probability weights

Once the sampling probabilities are computed, the weight \({w}_{k}\) assigned to subject \(k\) is:

In both typical and atypical NCC data, the weights of all cases have the same value, because the sampling probability is the same for all cases. The weights of controls depend on the control and are always larger (or equal) to 1, in order to represent multiple controls in the original cohort: the lower their probability of being sampled, the higher their weight [ 20 ].

Of note, the sum of the weights will generally not correspond to the number of subjects in the cohort, particularly if KM-type of weights are used. To compensate for this, the weights of controls should be rescaled [ 4 ]:

where \({w}_{k}{\prime}\) is the rescaled version of weight \({w}_{k}\) and \({n}_{c}\) is the total number of controls in the cohort.

Model performance metrics adjusted to the NCC data

As recommended in the TRIPOD guidelines for risk prediction models [ 23 ], models should be validated with respect to their discrimination ability, the agreement between observed and predicted outcomes (calibration), and the clinical utility they provide. Therefore, we focused on the following performance metrics to quantify these characteristics: C-index, threshold-based metrics (such as sensitivity and specificity), observed-to-events ratio, calibration slope, and decision curve analysis. The application of these metrics to full cohorts is straightforward, and we refer the reader to the literature for an in depth characterization of these metrics [ 24 , 25 ]. While these metrics can be obtained in NCC studies, they must be adjusted due to the under-sampling of controls, by using the weights of the subjects included in the NCC dataset. Of note, adjusted metrics should be used in both internal or external validation of the models in NCC datasets (see the section “ Bootstrapping and cross-validation in NCC data ”).

The C-index estimates the probability that the predicted order of the events of a randomly selected subject pair is correct. The C-index is calculated by identifying all possible pairs where at least one subject had an event (usable pairs). From these, concordant pairs and discordant pairs are identified as pairs where subjects with longer survival time have smaller or larger predicted risk, respectively. The C-index is then calculated as follows:

where \({C}_{k}\) and \({D}_{k}\) denote respectively the number of concordant and discordant pairs for subject \(k\) .

When a model is validated in an NCC dataset, the C-index should be adjusted by weighing the concordant and discordant pairs by the estimated weight of subject \(k\) ( \({w}_{k}\) ), as follows [ 4 ]:

Threshold-based performance metrics

Often, prediction models are used with decision thresholds: subjects above and below the risk threshold are considered high-risk and low-risk, respectively. Performance metrics such as sensitivity (SE), specificity (SP), positive predictive value (PPV) and negative predictive value (NPV) are informative to validate a model in this setting. The need to adjust PPV and NPV is clear, as these metrics are directly dependent on the prevalence of the outcome of interest in a cohort [ 26 ], but SE and SP must also be adjusted in NCC datasets, due to the biased sampling. Unbiased estimates of these metrics are obtained by using the weights of cases and controls as described in Table  2 .

The adjustment of these metrics has been described for model validation in NCC datasets when the outcome of interest is binary, and no survival data was available [ 6 ]. In this situation, the weights of all controls are the same and they correspond to the inverse of the control sampling fraction (i.e., \(\frac{1}{\frac{sampled \space controls}{total \space number \space of \space controls}}\) ). In Table  2 , we generalized this adjustment for survival data and for the possibility of controls having different weights.

Calibration

Model calibration aims to assess whether the absolute risks predicted by the model correctly estimate the observed risks. For example, for patients with, say, a 20% predicted risk of an event of interest, 20 patients out of 100 should indeed experience the event.

Calibration can be evaluated in multiple ways. We focused on the most important calibration metrics for risk prediction models [ 27 ]: mean calibration, calibration slope, and calibration plot.

Mean calibration corresponds to the difference between the proportion of observed events, for binary events; or 1-observed survival fraction at the chosen time point estimated by e.g. the Kaplan–Meier estimate, for survival data [ 28 ], and the average predicted risk. It can also be expressed as a ratio of observed to expected (i.e., predicted) events. In a NCC dataset, the observed events or survival fraction should be derived from the full cohort, and, apart from rounding differences, these should correspond to the weighted event proportion or weighted KM survival probability, respectively. Likewise, the average predicted risk should be computed using the weighted average of the predicted risk, with the previously described weights.

The calibration slope summarizes the strength of the association between predicted and observed outcomes [ 27 ]. It corresponds to the slope of the regression of the observed outcomes (binary or survival) on the linear predictor of the model. A perfectly calibrated model has a slope of 1, while lower and greater values indicate over- and underfitting respectively. Calibration can be visualized in a calibration plot showing the observed proportions or (1-observed survival fractions) against the average predicted risks for a given number of subject groups. For NCC datasets, a weighted regression of the observed outcomes (binary or survival) on the linear predictors of the model should be used to obtain the calibration slope (Supplementary Table  1 ). The calibration plot should show weighted observed proportions/survival fractions against weighted averages of the predicted risks for a given number of subject groups.

Decision curve analysis

Discrimination and calibration measures are insufficient to evaluate the model utility in the context of a clinical decision. Decision curve analysis aims to assess the net benefit of making clinical decisions based on the prediction model compared to other models or default strategies by accounting for the trade-off between the relative harms of false positives and false negatives [ 29 ]. The relative harm can be interpreted as how much worse it is to miss a poor clinical outcome and wrongly withholding treatment (i.e., false negative) compared to providing an unnecessary intervention to a patient who will not develop the outcome (i.e., false positive). For example, if the harm of missing a cancer is 4 times greater than performing an unnecessary invasive procedure, we can translate this relative harm into a probability threshold such that the patient should only undergo the procedure if the risk is greater than the threshold (1/(4 + 1) = 0.2). The decision curve is constructed by computing the net benefit over a clinically meaningful range of probability thresholds.

The net benefit (NB) at a given threshold is computed as follows [ 29 ]:

where \(TP\) denotes true positives, \(FP\) denotes false positives, \(n\) is the total number of subjects and \({p}_{t}\) is the probability threshold.

In NCC data, the net benefit of Eq. ( 6 ) should be calculated using the definitions of \(TP\) and \(FP\) for binary or survival data provided in Table  2 . Net benefit for binary outcomes in NCC data has already been derived in [ 30 ], but not for survival outcomes.

Bootstrapping and cross-validation in NCC data

Bootstrapping and cross-validation are techniques that are frequently employed to internally validate prediction models and to obtain estimates of the variability of model performance metrics. In general, bootstrap samples and cross-validation folds should mimic the original sampling design as well as possible [ 31 ]. Therefore, when these procedures are applied to NCC datasets, they should account for the fact that the NCC design is a stratified design, where controls are matched to cases (at least on follow-up time). This translates into obtaining bootstrap samples by sampling case–control pairs with replacement, rather than individual samples. In the case of cross-validation, case–control pairs should be included in the same fold in cross-validation schemes. Weighted metrics should also be used when evaluating the model within the NCC cross-validation folds and the bootstrap samples.

Code availability

We implemented a subset of the weighted performance metrics (threshold-based metrics, calibration plot and decision curve analysis for survival data). We used the R packages described in Supplementary Table  1 for computing sampling weights and the remaining weighted performance metrics. The code to reproduce our analysis is available at https://github.com/emc-dermatology/ncc-evaluation .

Real-world illustration: validation of the BOADICEA model

Breast cancer is the most commonly diagnosed cancer in women worldwide [ 32 ]. Identifying high-risk individuals is critical to reduce mortality and improve quality of life. However, excessive screening also results in false positives and overdiagnosis [ 33 ]. Risk prediction models can help healthcare systems by targeting women at high risk of developing breast cancer for screening, while reducing the side-effects of screening for those at lower risk.

In this clinical illustration, we focused on the validation of the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA version 5) model in the Rotterdam study, a population-based cohort. The performance of the model predicting the development of breast cancer (invasive or in situ) within 10 years has been previously evaluated in this cohort [ 18 ]. We use this cohort to show that it is also possible to validate a prediction model in a sub-cohort much smaller than the full cohort (more than 10 times smaller in this case), if the sub-cohort is carefully designed (i.e., the nested case–control dataset).

Model and study cohort

The BOADICEA model is an absolute risk prediction model that estimates the probability of developing breast cancer within 5 years, 10 years or within a lifetime, for women until the age of 80. The model uses genetic input variables such as a polygenic risk score (PRS) based on 313 breast cancer associated variants, together with nongenetic risk factors and family history. The nongenetic risk factors are: mammographic density, age at menarche, age at menopause, parity, age at first live birth, oral contraceptive use, hormonal treatment, height body mass index and alcohol intake. The formula for computing the absolute risk is available in Lee et al. [ 17 ]. The model was validated in the Rotterdam Study cohort, a population-based cohort of elderly Dutch individuals living in a district of Rotterdam in the Netherlands. In 2008, the cohort consisted of 14,926 subjects aged 45 years or older, out of which 8,823 were women. As in Lakeman et al. [ 18 ], we excluded all women for whom 10-year BOADICEA risk estimates could not be obtained: women for whom genotype data were not available or did not have enough quality ( n  = 2153), and women who, at the time of recruitment into the cohort, had breast cancer ( n  = 148) or were older than 70 years ( n  = 2145). This left a total of 4,377 women. All BOADICEA risk factors were available in this cohort, except for mammographic density. However, the BOADICEA model allows for missing information [ 17 ].

Ethics statement

The Rotterdam Study has been approved by the institutional review board (Medical Ethics Committee) of the Erasmus Medical Center and by the review board of The Netherlands Ministry of Health, Welfare and Sports. The analyses in this study were approved by the management team of the Rotterdam study.

Illustration setup

NCC datasets with one control per case were derived from the full cohort of 4,377 subjects in 3 different scenarios: a NCC design without any matching variables (NCC-NM); an NCC design with matching on an administrative variable (Rotterdam study sub-cohort), which is not strongly associated with the model predictions (NCC-MNR); and an NCC design with matching based on the non-genetic risk estimates (NCC-MR). In the NCC-MNR design, the matching variable has 3 categories, and in the NCC-MR, the matching variable has 6 categories, representing the risk percentages (from 1 to 6%). Control sampling in all designs was performed with incidence density sampling without duplicated subjects. Cases without a control after duplicate removal were matched to another control so that the total KM weight sum before rescaling was similar to the size of the full cohort. Control replacement is typically possible for NCC designs in rare outcomes and sufficiently loose matching.

The BOADICEA model was applied to both the full cohort and all the derived NCC datasets (Supplementary Fig.  1 ). Performance metrics, including the C-index, calibration metrics, and decision curve analysis, were computed in all cohorts. For the NCC datasets, the performance of the BOADICEA model was evaluated with and without adjustment for the NCC design. These unweighted (“naïve”) and weighted metrics were compared with the performance metrics on the full cohort. In practice, only one NCC dataset would be sampled, and performance metrics would be evaluated on that single cohort, however, to account for sampling variation, we repeated the derivation of NCC datasets 100 times in each scenario. As a sensitivity analysis, we used three ways to compute the sampling weights: Kaplan–Meier type of weights (KM), weights computed with logistic regression (GLM) and weights computed with generalized additive models (GAM). We also repeated the analyses with NCC datasets with 2 controls per case, to investigate the impact of the case–control ratio on the precision of the performance metric estimates.

A total of 4,377 subjects were included in our analysis, out of which 163 developed breast cancer within 10 years of their recruitment into the cohort (Table  3 ). The median follow-up was 12.9 (6.9–20.9) years for subjects without an event. Women who developed breast cancer (cases) differed significantly from those who did not (controls) in Body Mass Index and in their 313-variant PRS. The original Rotterdam study (RS-I) was extended twice (RS-II and RS-III) [ 34 ]. The proportion of cases was significantly different among these sub-cohorts.

Model validation in the full cohort

Similarly to the validation publication in the Rotterdam Study [ 18 ], the performance of the BOADICEA model was evaluated in all the subjects of this cohort, and compared to the performance of the risk estimates based on subsets of the model components (age only, age and non-genetic risk factors, and age and genetic component). All of the metrics are reported in Supplementary Table  2 ). The BOADICEA model (using age, non-genetic risk factors and the PRS) showed reasonable discriminative ability (C-index 0.65 (0.61–0.69)) and calibration (calibration slope 1.19 (0.87–1.54)). However, breast cancer occurrences were substantially underestimated, with an observed-to-expected ratio (O/E ratio) of 1.69 (1.42–1.93). This underestimation was more evident in the higher risk range (Supplementary Fig.  2 ). Despite the observed miscalibration, decision curve analysis showed that the BOADICEA model is still clinically useful for targeted screening in the vicinity of the outcome cumulative incidence (Fig.  3 ): for threshold probabilities between 3% and 7.5%, the model outperformed the treat all strategy. Interestingly, comparison of different subcomponents of the model shows that risk estimates based on the genetic component were more clinically useful than risk estimates relying on age or risk factors alone. Since risk predictions are often dichotomized to guide decisions, we used the risk threshold described in the BOADICEA development publication [ 17 ] (3%) and classified subjects with a risk prediction lower than 3% as low-risk and the remaining as high-risk. Performance metrics obtained with this risk threshold were suboptimal: namely, sensitivity to detect breast cancer occurrences was only 0.52 (0.43–0.61).

Decision curve analysis comparing different components of the BOADICEA model in the full cohort of 4377 women. The plot describes the decision curves for 1) the 10-year risk estimates based on Age alone (Age); 2) Age and Risk Factors (Age + RF); 3) Age and Polygenic Risk Score (Age + PRS); and, finally, for 4) the full BOADICEA model, with all these components (Age + RF + PRS). Risk factors do not include mammography density information

Model validation in NCC data

We evaluated the performance of the model in 3 scenarios with variations in the NCC sampling design; all sampling designs led to NCC datasets with 163 cases and 163 controls. To evaluate model predictions, we first analyzed performance metrics without setting any risk threshold. Fig.  4 clearly shows that the unweighted performance metrics obtained in the NCC datasets do not correspond to those obtained in the full cohort. The bias was larger when controls were matched to cases on a variable that is associated with the model predictions, but precision was similar for most metrics. In contrast, weighted performance metrics were unbiased in all scenarios. The results were very similar for other types of sampling weights (GLM and GAM weights, Supplementary Fig.  3 ). The unweighted calibration plot suggests that the model substantially underestimated the risk of the outcome (Fig.  5 ), while all weighted calibration plots resemble the calibration plot of the full cohort. Similarly, the unweighted decision curve indicates that the model is not more useful than the screen-all strategy, while the weighted decision curves are very similar to the decision curve on the full cohort (Fig.  6 ); therefore correctly showing that the model outperforms the screen-all strategy. Precision for lower probability ranges was slightly higher for the matched NCC datasets. However, for all designs the width of the confidence intervals and bias of net benefit estimates substantially increased with increasing risk thresholds, because there were fewer subjects with higher risk predictions, and therefore more variability across different realizations of the NCC sampling.

Performance metrics obtained in the full cohort and in the NCC datasets, for threshold-independent model predictions. The horizontal black line indicates the value of the performance metric in the full cohort and the dashed horizontal black lines indicate the upper and lower bounds of the 95% confidence interval of the performance metric in the full cohort. The color of the boxplots indicates whether performance metrics on the NCC datasets are weighted (“Yes”) or unweighted (“No”). NCC-NM: a regular NCC design with incidence density sampling and without any matching variables; NCC-MNR: a NCC design with incidence density sampling and matching on an administrative variable, which is not associated with the model predictions; NCC-MR: NCC design with incidence density sampling and matching based on the non-genetic risk predictions

Calibration plots for the BOADICEA model applied to NCC datasets. The full cohort and the NCC datasets were divided into 5 quantiles based on predicted risk probabilities. Event estimates of the full cohort are depicted in dark blue. Unweighted event estimates are depicted in salmon; weighted event estimates are depicted in light blue. Reported 95% confidence intervals were computed by considering the variance of the Kaplan–Meier estimates, and of the mean risk probability of each group within each NCC dataset, together with the variance of these estimates between all 100 samples of NCC datasets. NCC-NM: a regular NCC design with incidence density sampling and without any matching variables; NCC-MNR: an NCC design with incidence density sampling and matching on an administrative variable, which is not associated with the model predictions; NCC-MR: NCC design with incidence density sampling and matching based on the non-genetic risk predictions

Decision curves obtained for the BOADICEA model in the full and the NCC datasets. Unweighted net benefit in the NCC datasets is depicted in salmon; weighted net benefit is depicted in light blue. These net benefit estimates correspond to the average of the estimates obtained in the 100 samples of NCC datasets. Shaded areas correspond to the bootstrap-percentile 95% confidence interval obtained across all 100 samples of NCC datasets. The net benefit of the full cohort is depicted by the dashed dark blue line. The net benefit of screening everyone is depicted in black (“Treat all”), and the net benefit of screening no one is depicted in gray (“Treat none”). The net benefit of “Treat all” is unweighted in the unweighted plot and weighted in the remaining plots. NCC-NM: a regular NCC design with incidence density sampling and without any matching variables; NCC-MNR: an NCC design with incidence density sampling and matching on an administrative variable, which is not associated with the model predictions; NCC-MR: NCC design with incidence density sampling and matching based on the non-genetic risk predictions

As with the previous performance metrics, the threshold-dependent metrics (SE, SP, PPV and NPV) were biased without weighting, and close to unbiased when weighted (Fig.  7 ). Unbiased estimates were also obtained for all weighted metrics when 2 controls were sampled per case; however, the precision of performance estimates was higher for most performance metrics (Supplementary Table  3 , Supplementary Figs. 4 and 5 ). Lastly, model comparisons were also biased when unweighted metrics were used (Supplementary Fig.  6 ). Namely, using unweighted metrics to compare the BOADICEA model with the genetic component or without leads to a substantial underestimation of the benefit of the genetic component: the difference between the unweighted C-indexes of the two models was respectively 7% and 4% in the NCC-NM and NCC-MR scenarios, compared to 10% when the difference in C-index was computed using weighted metrics in either scenario, or based on the full cohort.

Threshold-based performance metrics obtained in the full cohort and in the NCC datasets. Here, the BOADICEA model was applied to the subjects, and those with a risk prediction lower than 3% were classified as low-risk. The horizontal black line indicates the value of the performance metric in the full cohort and the dashed horizontal black lines indicate the upper and lower bounds of the 95% confidence interval of the performance metric in the full cohort. The color of the boxplots indicates whether performance metrics of the NCC datasets are weighted (“Yes”) or unweighted (“No”). PPV: Positive predictive value, NPV: Negative predictive value, NCC-NM: a regular NCC design with incidence density sampling and without any matching variables; NCC-MNR: an NCC design with incidence density sampling and matching on an administrative variable, which is not associated with the model predictions; NCC-MR: NCC design with incidence density sampling and matching based on the non-genetic risk predictions

We systematically proposed how to validate prediction models in nested case–control data, by adjusting discriminative and calibration metrics, as well as calibration plots and decision curves. Despite their sample size being much smaller, we showed that NCC datasets can be used to obtain performance estimates that correspond to those of the original population-based cohorts, as long as performance metrics are appropriately weighted. The weighting procedure consists of estimating the sampling weight [ 20 ] of each subject in the NCC dataset and using those weights when computing each performance metric.

We illustrated the importance and validity of the described weighted metrics in a real-world case study where we validated the BOADICEA model in NCC datasets derived from the Rotterdam study. Although the BOADICEA model had previously been externally validated in the Rotterdam study [ 18 ], we showed that, if the model input variables had not been available for the entire cohort, unbiased performance estimates could have also been obtained in an NCC subset of the original cohort: the median/mean of the estimated metrics in the NCC datasets corresponded to their values in the Rotterdam study (Figs. 4 – 7 ).

NCC datasets are therefore a more cost-effective design for model validation, as they allow the estimation of unbiased performance metrics with a much smaller sample size: in this case less than 10% of the original cohort. This is of particular interest for evaluating the performance of models that require measuring biomarkers that are difficult to obtain in large cohorts, such as the polygenic risk scores included in the BOADICEA model. Of note, the smaller sample size of NCC datasets leads to higher uncertainty regarding the estimated performance metrics. If this is of concern, the number of controls sampled per case can be increased to increase precision of the estimates (Supplementary Table  3 , Supplementary Figs. 4 and 5 ).

We observed two distinct patterns depending on whether metrics directly depend on the outcome prevalence or not. Metrics such as the O/E ratio, net benefit, PPV and NPV are easily flagged as implausible if unweighted. For example, the unweighted calibration plot suggests that the model substantially underestimated the risk of the outcome (Fig.  5 ). On the other hand, metrics such as the C-index, calibration slope, sensitivity and specificity have plausible ranges even if unweighted.

However, we illustrated that using such metrics without weighting would lead to incorrect conclusions regarding model performance. In particular, we showed that when no weighting was applied, the C-index of the BOADICEA model was consistently underestimated in the NCC datasets. This underestimation has been observed in other studies 34,38 , and is, in fact, expected, as the NCC matching procedure (on time and on potential additional matching variables) frequently attenuates the association between the outcome and any model variables that might be associated with time/matching variables. This in turn decreases discriminative performance metrics. Conclusions regarding model calibration based on unweighted metrics would also be misleading: the unweighted calibration slope was lower than 1 in all scenarios, suggesting that the model was overfitted (Fig.  4 ), while the model was, in reality, underfitted. Furthermore, comparisons of the predictive ability of the different components of the model (non-genetic risk estimates, genetic risk estimates and combined model) would also lead to different conclusions if metrics are not weighted (Supplementary Fig.  6 ). Namely, in the NCC datasets matched based on risk factors, the improvement in predictive ability associated with adding the genetic component to the model was 4% based on unweighted C-indexes, compared to 10% based on the weighted or full cohort C-indexes. This observation is important as the NCC study design is frequently used to study the added value of biomarkers that are too expensive to collect in the full cohort. Our real-world example shows that it is essential to use weighted metrics to correctly estimate the improvement of performance metrics associated with novel biomarkers.

We have also shown that unbiased performance metrics can be obtained across different NCC sampling designs (with and without matching). This indicates that the effect of different types of biases introduced during cohort sampling can be mitigated during model validation by adjusting the performance metrics with sampling weights that account for the study design. Of note, while ignoring matching on administrative variables in the weight computation has a smaller impact than ignoring matching on variables associated with model risk predictions, matching on administrative variables should still be accounted for, if possible. Other studies that have investigated weighted metrics such as the C-index and calibration slope in matched NCC designs have shown biased estimation under fine matching [ 4 , 35 ]. We have not observed this in our study but the matching we employed was not as fine: there were only 3 categories when we matched based on entry into the Rotterdam study (NCC-MNR), and 6 categories when based on non-genetic risk estimates (NCC-MR), with subjects being well distributed among these categories. Moreover, Ganna et al. [ 4 ] evaluated a conditional logistic regression model that included the matching variables as predictors; meaning that they could not be included in the linear predictors used for risk estimation. This contributed to additional bias in the performance metrics obtained in the matched NCC design in their study. This was not the case with the BOADICEA model.

Finally, we have shown that unbiased performance metrics can be obtained using different methods for calculating the sampling weights (Supplementary Fig.  3 ), which means that weighted metrics are robust with respect to the computation method used for the weights. Similar conclusions regarding weight computation methods had already been demonstrated for the development of prediction models in NCC datasets [ 10 , 20 ], but not for their validation.

This study has some limitations. We illustrated the importance of using weighted performance metrics in a large real-world cohort, where using a nested case–control design can substantially reduce the cost of validating a prediction model that requires expensive data collection. However, the use of a real-world dataset prevents us from studying the robustness of our conclusions to variations in parameters such as outcome incidence and the magnitude of the performance metrics of the model. A simulation study [ 35 ], which investigated this for a subset of the performance metrics, showed that the C-index was consistent for different outcome incidence rates, but calibration metrics deteriorated for lower incidence rates (5%). This slightly contrasts with our study, where the outcome cumulative incidence was 4.4% (95% CI 3.7–5.1%) and all metrics could be unbiasedly estimated. However, we focused on external validation of the model while Lee et al. [ 35 ] reported fivefold cross-validation metrics for the calibration metrics which might partially explain their observed decline in performance for lower outcome incidence.

We have extensively covered the most commonly recommended metrics to assess the discrimination, calibration and clinical utility of prediction models [ 23 ]; however, there are other metrics and extensions that could be used. For example, goodness-of-fit tests such as Grønnesby and Borgan or Hosmer and Lemeshow’s test are employed to assess model calibration, although due to their limitations they are falling into disuse. The use of such tests in NCC datasets has been described in the literature [ 4 , 8 ]. Quantification of the incremental value of additional predictors can be estimated with the net reclassification index, and a weighted version is provided in [ 4 ]. However, concerns have also been raised regarding this metric [ 36 ].

Furthermore, for simplicity, we did not account for competing risks in our real-world illustration, as this would require using performance metrics that are adjusted to the competing risk scenario [ 37 ], and modifying the sampling procedure of the NCC study to account for competing risks [ 38 ]. In fact, with strong competing risks or presence of multiple outcomes, the case-cohort design is preferred to the NCC design. Weighting of performance metrics in this sampling design is also needed, although recommended methods to compute sampling weights in this design [ 39 ] are different than the ones described in our study.

Despite the above limitations, the described weighted metrics should be relevant for a wide audience (e.g., clinicians, machine learning practitioners, epidemiologists and biostatisticians), as they must be used for a correct performance evaluation of any type of model that estimates risk probabilities in an NCC study, from the widely used Cox model to machine learning or deep learning methods. Furthermore, difficulties in model performance evaluation, which resemble the ones we discussed, are addressed in the literature under different names (such as covariate shift in the machine learning field [ 40 ]). The NCC sampling design is a special case within these concepts, as it performs sampling in a cost-effective way.

In summary, this study provides clear guidance on how prediction models should be validated in NCC studies using relevant performance metrics. Previously, most of this information was scattered in the literature and not available for all the metrics. Namely, weighted versions of performance metrics are available in different R packages [ 8 ], but we had to implement weight adjustments for threshold-based metrics and decision curve analyses for survival data. The code to implement the adjustment of these performance metrics with sampling weights is now available in a GitHub repository (see section “ code availability ”) for easy implementation and should facilitate the adoption of weighted metrics also by non-specialists.

Clinical prediction models can be validated in NCC studies if the performance metrics are appropriately adjusted using the sampling weights of the subjects in the NCC dataset. If performance metrics are not weighted, their estimates will be biased, with the magnitude of the bias being higher when matching variables are correlated with the model predictions. The choice of the method to compute sampling weights does not lead to large changes in the estimated weighted metrics, as long as the sampling design and all matching variables are considered in the computation. These results are particularly relevant for the validation of models that predict rare outcomes, and whose input variables cannot be measured in all subjects in the validation cohort.

Availability of data and materials

The dataset analyzed in this study is not publicly available due to privacy regulations; however data can be obtained upon request to the management team of the Rotterdam Study ([email protected]).

Abbreviations

Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm

Confidence Interval

Generalized additive model

Generalized linear model

Kaplan–Meier

Net benefit

Nested case–control

An NCC design with incidence density sampling and matching on an administrative variable

An NCC design with incidence density sampling and matching based on the non-genetic risk estimates

Regular NCC design with incidence density sampling and without any matching variables

Negative predictive value

Observed-to-expected events ratio

Positive predictive value

Polygenic risk score

Risk factors

Sensitivity

Specificity

McCarthy CE, Bonnet LJ, Marcus MW, Field JK. Development and validation of a multivariable risk prediction model for head and neck cancer using the UK Biobank. Int J Oncol. 2020;57(5):1192–202.

CAS   PubMed   Google Scholar  

Cederholm J, Eeg-Olofsson K, Eliasson B, Zethelius B, Nilsson PM, Gudbjörnsdottir S. Risk prediction of cardiovascular disease in type 2 diabetes: a risk equation from the Swedish National Diabetes Register. Diabetes Care. 2008;31(10):2038–43.

Article   PubMed   PubMed Central   Google Scholar  

Moons KGM, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98(9):683–90.

Article   PubMed   Google Scholar  

Ganna A, Reilly M, De Faire U, Pedersen N, Magnusson P, Ingelsson E. Risk prediction measures for case-cohort and nested case-control designs: An application to cardiovascular disease. Am J Epidemiol. 2012;175(7):715–24.

Salim A, Delcoigne B, Villaflores K, Koh WP, Yuan JM, van Dam RM, et al. Comparisons of risk prediction methods using nested case-control data. Stat Med. 2017;36(3):455–65.

Biesheuvel CJ, Vergouwe Y, Oudega R, Hoes AW, Grobbee DE, Moons KGM. Advantages of the nested case-control design in diagnostic research. BMC Med Res Methodol. 2008;8(1):1–7.

Article   Google Scholar  

Moons KGM, Van Klei W, Kalkman CJ. Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia. Anesth Analg. 2003;96(6):1843–4.

Choudhury PP, Maas P, Wilcox A, Wheeler W, Brook M, Check D, et al. iCARE: An R package to build, validate and apply absolute risk models. PLoS ONE. 2020;15(2):e0228198.

Kim RS. Analysis of Nested Case-Control Study Designs: Revisiting the Inverse Probability Weighting Method. Communications for Statistical Applications and Methods. 2013;20(6):455–66.

Borgan Ø, Keogh R. Nested case–control studies: should one break the matching? Lifetime Data Anal. 2015;21(4):517–41.

Zelic R, Zugna D, Bottai M, Andrén O, Fridfeldt J, Carlsson J, et al. Estimation of Relative and Absolute Risks in a Competing-Risks Setting Using a Nested Case-Control Study Design: Example from the ProMort Study. Am J Epidemiol. 2019;188(6):1165–73.

Delcoigne B, Colzani E, Prochazka M, Gagliardi G, Hall P, Abrahamowicz M, et al. Breaking the matching in nested case–control data offered several advantages for risk estimation. J Clin Epidemiol. 2017;1(82):79–86.

Rentroia-Pacheco B, Tokez S, Bramer EM, Venables ZC, Van De Werken HJG, Bellomo D, et al. Personalised decision making to predict absolute metastatic risk in cutaneous squamous cell carcinoma: development and validation of a clinico-pathological model. eClinicalMedicine. 2023;63:102150.

Murphy JD, Olshan AF, Lin FC, Troester MA, Nichols HB, Butt J, et al. A Predictive Model of Noncardia Gastric Adenocarcinoma Risk Using Antibody Response to Helicobacter pylori Proteins and Pepsinogen. Cancer Epidemiol Biomark Prev. 2022;31(4):811–20.

Article   CAS   Google Scholar  

Hoogeveen RM, Pereira JPB, Nurmohamed NS, Zampoleri V, Bom MJ, Baragetti A, et al. Improved cardiovascular risk prediction using targeted plasma proteomics in primary prevention. Eur Heart J. 2020;41(41):3998–4007.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Reps JM, Ryan PB, Rijnbeek PR, Schuemie MJ. Design matters in patient-level prediction: evaluation of a cohort vs. case-control design when developing predictive models in observational healthcare datasets. Journal of Big Data. 2021;8(1):1–18.

Lee A, Mavaddat N, Wilcox AN, Cunningham AP, Carver T, Hartley S, et al. BOADICEA: a comprehensive breast cancer risk prediction model incorporating genetic and nongenetic risk factors. Genet Med. 2019;21(8):1708–18.

Lakeman IMM, Rodríguez-Girondo M, Lee A, Ruiter R, Stricker BH, Wijnant SRA, et al. Validation of the BOADICEA model and a 313-variant polygenic risk score for breast cancer risk prediction in a Dutch prospective cohort. Genet Med. 2020;22(11):1803–11.

Zhou QM, Wang X, Zheng Y, Cai T. New weighting methods when cases are only a subset of events in a nested case-control study. Biom J. 2022;64(7):1240–59.

Støer NC, Samuelsen SO. MultipleNCC: Inverse probability weighting of nested case-control data. R Journal. 2016;8(2):5–18.

Langholz B, Richardson D. Are Nested Case-Control Studies Biased? Epidemiology. 2009;20(3):321–9.

Støer NC, Samuelsen SO. Comparison of estimators in nested case-control studies with multiple outcomes. Lifetime Data Anal. 2012;18(3):261–83.

Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): Explanation and elaboration. Ann Intern Med. 2015;162(1):W1–73.

Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating Second Edition. New York: NY: Springer Nature Switzerland; 2019. (Statistics for Biology and Health).

Book   Google Scholar  

Vickers AJ, Elkin EB. Decision curve analysis: A novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565–74.

Parikh R, Mathai A, Parikh S, Sekhar GC, Thomas R. Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol. 2008;56(1):45–50.

Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: From utopia to empirical data. J Clin Epidemiol. 2016;1(74):167–76.

McLernon DJ, Giardiello D, Van Calster B, Wynants L, van Geloven N, van Smeden M, et al. Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models. Ann Intern Med. 2023;176(1):105–14.

Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Medical Inform Decis Mak. 2008;8:53.

Pfeiffer RM, Gail MH. Estimating the decision curve and its precision from three study designs. Biom J. 2020;62(3):764–76.

Wieczorek J, Guerin C, McMahon T. K-fold cross-validation for complex sample surveys. In: In: Stat. John Wiley & Sons, Ltd; 2022. p. e454.

Google Scholar  

Arnold M, Morgan E, Rumgay H, Mafra A, Singh D, Laversanne M, et al. Current and future burden of breast cancer: Global statistics for 2020 and 2040. Breast. 2022;1(66):15–23.

Marmot M, Altman DG, Cameron DA, Dewar JA, Thompson SG, Wilcox M. The benefits and harms of breast cancer screening: an independent review. The Lancet. 2012;380(9855):1778–86.

Ikram MA, Brusselle G, Ghanbari M, Goedegebure A, Ikram MK, Kavousi M, et al. Objectives, design and main findings until 2020 from the Rotterdam Study. European J Epidemiol. 2020;35(5):483–517.

Lee M, Zeleniuch-Jacquotte A, Liu M. Empirical evaluation of sub-cohort sampling designs for risk prediction modeling. J Appl Stat. 2021;48(8):1374–401.

Pepe MS, Fan J, Feng Z, Gerds T, Hilden J. The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets. Stat Biosci. 2015;7(2):282–95.

Ramspek CL, Teece L, Snell KIE, Evans M, Riley RD, Van Smeden M, et al. Lessons learnt when accounting for competing events in the external validation of time-To-event prognostic models. Int J Epidemiol. 2022;51(2):615–25.

Wolkewitz M, Cooper BS, Palomar-Martinez M, Olaechea-Astigarraga P, Alvarez-Lerma F, Schumacher M. Nested case-control studies in cohorts with competing events. Epidemiology. 2014;25(1):122–5.

Sanderson J, Thompson SG, White IR, Aspelund T, Pennells L. Derivation and assessment of risk prediction models using case-cohort data. BMC Med Res Methodol. 2013;13:113.

Sugiyama M, Krauledat M, Müller KR. Covariate shift adaptation by importance weighted cross validation. J Mach Learn Res. 2007;8:985–1005.

Download references

Acknowledgements

The authors are grateful to the study participants, the staff from the Rotterdam Study and the participating general practitioners and pharmacists. We would like to thank Lara Pozza for her valuable comments on the manuscript.

PPP Allowance made available by Health ~ Holland, Top Sector Life Sciences & Health, to stimulate public–private partnerships. The funder had no role in the study design, data collection, analysis, interpretation or writing of the manuscript.

Author information

Loes M. Hollestein and David van Klaveren jointly supervised the work.

Authors and Affiliations

Department of Dermatology, Erasmus Medical Center Cancer Institute, Erasmus University Medical Center, Dr. Molewaterplein 40, Rotterdam, 3015 GD, The Netherlands

Barbara Rentroia-Pacheco, Marlies Wakkee & Loes M. Hollestein

SkylineDx B.V, Rotterdam, The Netherlands

Domenico Bellomo

Department of Human Genetics, Leiden University Medical Center, Leiden, The Netherlands

Inge M. M. Lakeman

Department of Clinical Genetics, Leiden University Medical Center, Leiden, The Netherlands

Department of Research, Netherlands Comprehensive Cancer Organization (IKNL), Utrecht, The Netherlands

Loes M. Hollestein

Department of Public Health, Center for Medical Decision Making, Erasmus University Medical Center, Rotterdam, The Netherlands

David van Klaveren

You can also search for this author in PubMed   Google Scholar

Contributions

BRP, LH and DvK conceived and designed the study. BRP performed the analyses. IMML contributed to data collection and interpretation. BRP wrote the first draft of the manuscript. DB, MW, LH and DvK supervised the study. All authors interpreted the data and critically reviewed the manuscript. All authors accept responsibility for the decision to submit this publication.

Corresponding author

Correspondence to Barbara Rentroia-Pacheco .

Ethics declarations

Ethics approval and consent to participate.

The Rotterdam Study complies with the Declaration of Helsinki and has been approved by the institutional review board (Medical Ethics Committee) of the Erasmus Medical Center and by the review board of The Netherlands Ministry of Health, Welfare and Sports. All participants provided written informed consent to participate in the study.

Consent for publication

Not applicable.

Competing interests

DB is employed by SkylineDx. Other authors have no conflict of interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Rentroia-Pacheco, B., Bellomo, D., Lakeman, I.M.M. et al. Weighted metrics are required when evaluating the performance of prediction models in nested case–control studies. BMC Med Res Methodol 24 , 115 (2024). https://doi.org/10.1186/s12874-024-02213-6

Download citation

Received : 14 July 2023

Accepted : 04 April 2024

Published : 17 May 2024

DOI : https://doi.org/10.1186/s12874-024-02213-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Prediction model validation
• Nested case–control study
• Rare outcomes
• Weighted metrics

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Med Libr Assoc
• v.107(1); 2019 Jan

Distinguishing case study as a research method from case reports as a publication type

The purpose of this editorial is to distinguish between case reports and case studies. In health, case reports are familiar ways of sharing events or efforts of intervening with single patients with previously unreported features. As a qualitative methodology, case study research encompasses a great deal more complexity than a typical case report and often incorporates multiple streams of data combined in creative ways. The depth and richness of case study description helps readers understand the case and whether findings might be applicable beyond that setting.

Single-institution descriptive reports of library activities are often labeled by their authors as “case studies.” By contrast, in health care, single patient retrospective descriptions are published as “case reports.” Both case reports and case studies are valuable to readers and provide a publication opportunity for authors. A previous editorial by Akers and Amos about improving case studies addresses issues that are more common to case reports; for example, not having a review of the literature or being anecdotal, not generalizable, and prone to various types of bias such as positive outcome bias [ 1 ]. However, case study research as a qualitative methodology is pursued for different purposes than generalizability. The authors’ purpose in this editorial is to clearly distinguish between case reports and case studies. We believe that this will assist authors in describing and designating the methodological approach of their publications and help readers appreciate the rigor of well-executed case study research.

Case reports often provide a first exploration of a phenomenon or an opportunity for a first publication by a trainee in the health professions. In health care, case reports are familiar ways of sharing events or efforts of intervening with single patients with previously unreported features. Another type of study categorized as a case report is an “N of 1” study or single-subject clinical trial, which considers an individual patient as the sole unit of observation in a study investigating the efficacy or side effect profiles of different interventions. Entire journals have evolved to publish case reports, which often rely on template structures with limited contextualization or discussion of previous cases. Examples that are indexed in MEDLINE include the American Journal of Case Reports , BMJ Case Reports, Journal of Medical Case Reports, and Journal of Radiology Case Reports . Similar publications appear in veterinary medicine and are indexed in CAB Abstracts, such as Case Reports in Veterinary Medicine and Veterinary Record Case Reports .

As a qualitative methodology, however, case study research encompasses a great deal more complexity than a typical case report and often incorporates multiple streams of data combined in creative ways. Distinctions include the investigator’s definitions and delimitations of the case being studied, the clarity of the role of the investigator, the rigor of gathering and combining evidence about the case, and the contextualization of the findings. Delimitation is a term from qualitative research about setting boundaries to scope the research in a useful way rather than describing the narrow scope as a limitation, as often appears in a discussion section. The depth and richness of description helps readers understand the situation and whether findings from the case are applicable to their settings.

CASE STUDY AS A RESEARCH METHODOLOGY

Case study as a qualitative methodology is an exploration of a time- and space-bound phenomenon. As qualitative research, case studies require much more from their authors who are acting as instruments within the inquiry process. In the case study methodology, a variety of methodological approaches may be employed to explain the complexity of the problem being studied [ 2 , 3 ].

Leading authors diverge in their definitions of case study, but a qualitative research text introduces case study as follows:

Case study research is defined as a qualitative approach in which the investigator explores a real-life, contemporary bounded system (a case) or multiple bound systems (cases) over time, through detailed, in-depth data collection involving multiple sources of information, and reports a case description and case themes. The unit of analysis in the case study might be multiple cases (a multisite study) or a single case (a within-site case study). [ 4 ]

Methodologists writing core texts on case study research include Yin [ 5 ], Stake [ 6 ], and Merriam [ 7 ]. The approaches of these three methodologists have been compared by Yazan, who focused on six areas of methodology: epistemology (beliefs about ways of knowing), definition of cases, design of case studies, and gathering, analysis, and validation of data [ 8 ]. For Yin, case study is a method of empirical inquiry appropriate to determining the “how and why” of phenomena and contributes to understanding phenomena in a holistic and real-life context [ 5 ]. Stake defines a case study as a “well-bounded, specific, complex, and functioning thing” [ 6 ], while Merriam views “the case as a thing, a single entity, a unit around which there are boundaries” [ 7 ].

Case studies are ways to explain, describe, or explore phenomena. Comments from a quantitative perspective about case studies lacking rigor and generalizability fail to consider the purpose of the case study and how what is learned from a case study is put into practice. Rigor in case studies comes from the research design and its components, which Yin outlines as (a) the study’s questions, (b) the study’s propositions, (c) the unit of analysis, (d) the logic linking the data to propositions, and (e) the criteria for interpreting the findings [ 5 ]. Case studies should also provide multiple sources of data, a case study database, and a clear chain of evidence among the questions asked, the data collected, and the conclusions drawn [ 5 ].

Sources of evidence for case studies include interviews, documentation, archival records, direct observations, participant-observation, and physical artifacts. One of the most important sources for data in qualitative case study research is the interview [ 2 , 3 ]. In addition to interviews, documents and archival records can be gathered to corroborate and enhance the findings of the study. To understand the phenomenon or the conditions that created it, direct observations can serve as another source of evidence and can be conducted throughout the study. These can include the use of formal and informal protocols as a participant inside the case or an external or passive observer outside of the case [ 5 ]. Lastly, physical artifacts can be observed and collected as a form of evidence. With these multiple potential sources of evidence, the study methodology includes gathering data, sense-making, and triangulating multiple streams of data. Figure 1 shows an example in which data used for the case started with a pilot study to provide additional context to guide more in-depth data collection and analysis with participants.

Key sources of data for a sample case study

VARIATIONS ON CASE STUDY METHODOLOGY

Case study methodology is evolving and regularly reinterpreted. Comparative or multiple case studies are used as a tool for synthesizing information across time and space to research the impact of policy and practice in various fields of social research [ 9 ]. Because case study research is in-depth and intensive, there have been efforts to simplify the method or select useful components of cases for focused analysis. Micro-case study is a term that is occasionally used to describe research on micro-level cases [ 10 ]. These are cases that occur in a brief time frame, occur in a confined setting, and are simple and straightforward in nature. A micro-level case describes a clear problem of interest. Reporting is very brief and about specific points. The lack of complexity in the case description makes obvious the “lesson” that is inherent in the case; although no definitive “solution” is necessarily forthcoming, making the case useful for discussion. A micro-case write-up can be distinguished from a case report by its focus on briefly reporting specific features of a case or cases to analyze or learn from those features.

DATABASE INDEXING OF CASE REPORTS AND CASE STUDIES

Disciplines such as education, psychology, sociology, political science, and social work regularly publish rich case studies that are relevant to particular areas of health librarianship. Case reports and case studies have been defined as publication types or subject terms by several databases that are relevant to librarian authors: MEDLINE, PsycINFO, CINAHL, and ERIC. Library, Information Science & Technology Abstracts (LISTA) does not have a subject term or publication type related to cases, despite many being included in the database. Whereas “Case Reports” are the main term used by MEDLINE’s Medical Subject Headings (MeSH) and PsycINFO’s thesaurus, CINAHL and ERIC use “Case Studies.”

Case reports in MEDLINE and PsycINFO focus on clinical case documentation. In MeSH, “Case Reports” as a publication type is specific to “clinical presentations that may be followed by evaluative studies that eventually lead to a diagnosis” [ 11 ]. “Case Histories,” “Case Studies,” and “Case Study” are all entry terms mapping to “Case Reports”; however, guidance to indexers suggests that “Case Reports” should not be applied to institutional case reports and refers to the heading “Organizational Case Studies,” which is defined as “descriptions and evaluations of specific health care organizations” [ 12 ].

PsycINFO’s subject term “Case Report” is “used in records discussing issues involved in the process of conducting exploratory studies of single or multiple clinical cases.” The Methodology index offers clinical and non-clinical entries. “Clinical Case Study” is defined as “case reports that include disorder, diagnosis, and clinical treatment for individuals with mental or medical illnesses,” whereas “Non-clinical Case Study” is a “document consisting of non-clinical or organizational case examples of the concepts being researched or studied. The setting is always non-clinical and does not include treatment-related environments” [ 13 ].

Both CINAHL and ERIC acknowledge the depth of analysis in case study methodology. The CINAHL scope note for the thesaurus term “Case Studies” distinguishes between the document and the methodology, though both use the same term: “a review of a particular condition, disease, or administrative problem. Also, a research method that involves an in-depth analysis of an individual, group, institution, or other social unit. For material that contains a case study, search for document type: case study.” The ERIC scope note for the thesaurus term “Case Studies” is simple: “detailed analyses, usually focusing on a particular problem of an individual, group, or organization” [ 14 ].

PUBLICATION OF CASE STUDY RESEARCH IN LIBRARIANSHIP

We call your attention to a few examples published as case studies in health sciences librarianship to consider how their characteristics fit with the preceding definitions of case reports or case study research. All present some characteristics of case study research, but their treatment of the research questions, richness of description, and analytic strategies vary in depth and, therefore, diverge at some level from the qualitative case study research approach. This divergence, particularly in richness of description and analysis, may have been constrained by the publication requirements.

As one example, a case study by Janke and Rush documented a time- and context-bound collaboration involving a librarian and a nursing faculty member [ 15 ]. Three objectives were stated: (1) describing their experience of working together on an interprofessional research team, (2) evaluating the value of the librarian role from librarian and faculty member perspectives, and (3) relating findings to existing literature. Elements that signal the qualitative nature of this case study are that the authors were the research participants and their use of the term “evaluation” is reflection on their experience. This reads like a case study that could have been enriched by including other types of data gathered from others engaging with this team to broaden the understanding of the collaboration.

As another example, the description of the academic context is one of the most salient components of the case study written by Clairoux et al., which had the objectives of (1) describing the library instruction offered and learning assessments used at a single health sciences library and (2) discussing the positive outcomes of instruction in that setting [ 16 ]. The authors focus on sharing what the institution has done more than explaining why this institution is an exemplar to explore a focused question or understand the phenomenon of library instruction. However, like a case study, the analysis brings together several streams of data including course attendance, online material page views, and some discussion of results from surveys. This paper reads somewhat in between an institutional case report and a case study.

The final example is a single author reporting on a personal experience of creating and executing the role of research informationist for a National Institutes of Health (NIH)–funded research team [ 17 ]. There is a thoughtful review of the informationist literature and detailed descriptions of the institutional context and the process of gaining access to and participating in the new role. However, the motivating question in the abstract does not seem to be fully addressed through analysis from either the reflective perspective of the author as the research participant or consideration of other streams of data from those involved in the informationist experience. The publication reads more like a case report about this informationist’s experience than a case study that explores the research informationist experience through the selection of this case.

All of these publications are well written and useful for their intended audiences, but in general, they are much shorter and much less rich in depth than case studies published in social sciences research. It may be that the authors have been constrained by word counts or page limits. For example, the submission category for Case Studies in the Journal of the Medical Library Association (JMLA) limited them to 3,000 words and defined them as “articles describing the process of developing, implementing, and evaluating a new service, program, or initiative, typically in a single institution or through a single collaborative effort” [ 18 ]. This definition’s focus on novelty and description sounds much more like the definition of case report than the in-depth, detailed investigation of a time- and space-bound problem that is often examined through case study research.

Problem-focused or question-driven case study research would benefit from the space provided for Original Investigations that employ any type of quantitative or qualitative method of analysis. One of the best examples in the JMLA of an in-depth multiple case study that was authored by a librarian who published the findings from her doctoral dissertation represented all the elements of a case study. In eight pages, she provided a theoretical basis for the research question, a pilot study, and a multiple case design, including integrated data from interviews and focus groups [ 19 ].

We have distinguished between case reports and case studies primarily to assist librarians who are new to research and critical appraisal of case study methodology to recognize the features that authors use to describe and designate the methodological approaches of their publications. For researchers who are new to case research methodology and are interested in learning more, Hancock and Algozzine provide a guide [ 20 ].

We hope that JMLA readers appreciate the rigor of well-executed case study research. We believe that distinguishing between descriptive case reports and analytic case studies in the journal’s submission categories will allow the depth of case study methodology to increase. We also hope that authors feel encouraged to pursue submitting relevant case studies or case reports for future publication.

Editor’s note: In response to this invited editorial, the Journal of the Medical Library Association will consider manuscripts employing rigorous qualitative case study methodology to be Original Investigations (fewer than 5,000 words), whereas manuscripts describing the process of developing, implementing, and assessing a new service, program, or initiative—typically in a single institution or through a single collaborative effort—will be considered to be Case Reports (formerly known as Case Studies; fewer than 3,000 words).

Cookies on GOV.UK

We use some essential cookies to make this website work.

We’d like to set additional cookies to understand how you use GOV.UK, remember your settings and improve government services.

We also use cookies set by other sites to help us deliver content from their services.

You have accepted additional cookies. You can change your cookie settings at any time.

You have rejected additional cookies. You can change your cookie settings at any time.

• Education, training and skills
• Further and higher education, skills and vocational training
• Sector-based Work Academy Programme: qualitative case study research
• Department for Work & Pensions

SWAP Qualitative Case Study Research: Annexes

Updated 16 May 2024

© Crown copyright 2024

This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: [email protected] .

Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned.

This publication is available at https://www.gov.uk/government/publications/sector-based-work-academy-programme-qualitative-case-study-research/swap-qualitative-case-study-research-annexes

Annex 1: SWAP Theory of Change Logic Model

The flow diagram presents the following information:.

Inputs mainly under the heading ‘Government (Continued benefits, training costs, barriers to participation, analytical resources)’:

‘Employer -Time for set up/management/ feedback’

‘Training Provider - Time for set up/management’

‘Employer Advisor’, ‘Work Coach’

These inputs flow into the following activities:

• ‘Local Labour market analysis’
• ‘Arrange SWAP ’
• ‘Disseminate available SWAP with JCP ’
• ‘Awareness of available SWAP ’
• ‘Sell SWAP to claimant’
• ‘Refer claimant’

These activities flow to the output: ‘Claimant agrees to participate in SWAP ’. The claimant agreement flows on to the claimant input: ‘Investment of time and initial travel costs (reimbursable)’ and the following work coach activities:

• ‘Barrier Assessment’
• ‘Record referral’

Then the following employer activities:

• ‘Background checks (if applicable)’
• ‘Workplace adjustment (if applicable)’ 

These all flow into the output ‘Claimant starts SWAP ’.

The main SWAP portion of the theory of change starts here with the activities:

• ‘Pre-employment Training’ (with side flow to ‘Claimant gains certification (if applicable)’)
• ‘Work placement’
• ‘Guaranteed Interview’

The main SWAP activities flow to the following three possible outputs:

• ‘Claimant completes SWAP : Interview successful’
• ‘Claimant completes SWAP : Interview unsuccessful’
• ‘Claimant does not complete SWAP ’)

Regardless of which output, the diagram shows a flow to the activity ‘WC completes SWAP Tracker’. 

If Claimant completes SWAP with an unsuccessful interview, or the Claimant does not complete the SWAP , these outputs flow into the short term outcome of ‘additional needs identified’. The unsuccessful interview also flows into the short term outcome ‘interview feedback from employer’.

Both of these short term outcomes flow into the activity ‘Reflection with WC’ and this can flow into the activity ‘Claimant applies for other jobs in sector’. Claimant applies for other jobs in sector can flow into a successful short term outcome ‘claimant gains employment in new sector’ or an unsuccessful short term outcome, in which case ‘claimant reengages with WC’. 

Claimants unsuccessful at SWAP or subsequent interviews (through engagement with WC activities) flow into either:

• the short-term outcome ‘Increased employability’
• the medium term outcome ‘improve claimants employability’.

Collectively the short and medium term outcomes flow into the impact ‘Value for unsuccessful claimants’. 

If the claimant is successful at either the guaranteed or subsequent interview, these flow into the output ‘ Claimant enters work in new sector’ which flows onto the following four short-term outcomes:

• ‘Change in attitude towards working in new sector’
• ‘Claimant has skills to succeed at new job’
• ‘Increased earning’
• ‘Reduced UC /benefit’

This may flow into the following medium term outcomes:

• ‘ UC ends (or is maintained at reduced level)’
• ‘Sustained employment (18 months)’
• ‘Career progression’

These medium term outcomes flow into the impacts:

• ‘Increased employment’
• ‘Reduced UC costs’

If the claimant is successful at the guaranteed interview, and subject to the assumption ‘Employer outputs and outcomes are dependent on the SWAP meeting employer expectations’, the following employer outputs are recorded:

• ‘Reduced vacancies’
• ‘Employer social responsibility goals met’
• ‘Development of local workforce’ 

‘Reduced vacancies’ and ‘Employer social responsibility goals met’ flow to the short-term outcome ‘Employer satisfied with SWAP experience’. This then flows to the activity ‘ DWP gathers feedback from claimants and employers’ and the short-term outcome ‘Collated employer success stories’.

These short-term outcomes flow to the medium term outcomes:

• ‘Businesses return for additional SWAP ’
• ‘Increased employer uptake in SWAP or other provisions’
• ‘Improved Attitudes towards hiring DWP claimants’
• ‘Employers approaching DWP with vacancies more readily’ 

‘Development of local workforce’ output flows to the following short-term outcomes:

• ‘ SWAP aligns with local market need including sector shortages’
• ‘Sector pathways identified’
• ‘Change in attitude towards working in new sector’, which flow to the medium term outcome ‘Improved fit between employers and claimants’. 

Medium term outcomes in this employer focussed part of the theory of change flow to the impacts ‘Improved DWP relationship with business sector’ and ‘increased employment’.

Annex 2: Participant characteristics

Table 3: employer, training provider and claimant participants by swap sector, table 4: claimant participant characteristics, annex 3: additional methodology details.

This annex includes additional information about how the case study research was conducted.

Contacting claimants

A random sample of 150 claimants was drawn for each case study area (600 claimants in total across the four areas) in order to achieve 10 claimant interviews in each district. This sample size was in line with previous, similar research (in terms of mode, length and recruitment approach), which achieved a response rate of approximately 1 in 15 claimants. The sample was sourced from the SWAP manual trackers completed by each district which detail which claimants are referred each week to the programme. Claimant identification numbers were then linked to centrally held contact information (for example, postal address and telephone number).

The stratification of the sample was limited by the quality of data DWP holds on certain claimant characteristics (for example, ethnicity and disability information was not available) as well as claimants’ SWAP journey (only claimant start dates on the pre-employment training ( PET ) were consistently recorded by all areas). As a result, it was impossible to identify in advance claimants who had dropped out of a SWAP part-way through, or claimants who were successful at the guaranteed interview stage, which limited the study’s ability to explore these aspects in detail. The sample drawn was, therefore, broadly reflective (rather than representative) of the claimant population who started on a SWAP in terms of gender and sector of SWAP , and consisted of individuals who had started the SWAP PET within the previous 12 weeks of the sample being drawn. This time period was agreed in order to ensure the feasibility of obtaining a sample of 150 claimants from each area, while minimising as much as possible the risk of recall bias within claimant accounts of their experience.

It is important to note the claimant sample was delivered in two separate stages, to reflect the gap in fieldwork between Area 2 and Area 3. The samples for Areas 3 and 4, was additionally stratified by age (18 to 24 years vs. 25+ years) to account for the small number of potential participants aged 18 to 24 years provided in the sample for Areas 1 and 2.

All claimants in the sample were sent an advance letter to the address held on DWP ’s central records. This letter provided further information about the research, what their participation would involve, data processing information and an email address to which they could write if they wanted to opt-out. Claimants were called using the telephone numbers provided in each sample, and while formal quotas for recruiting participants were not used, calls were targeted to achieve a spread in terms of claimant gender, age and sector of SWAP (the latter was obtained from the SWAP manual trackers and was therefore dependent on DWP staff interpretations of this at the local level). Claimants were called up to three times without a response before they were not contacted any further. During the calls, researchers emphasised their independence from benefits processing and that decisions regarding participation would not affect claimants’ benefits in any way. Each interview lasted approximately 30 to 45 minutes and claimants received a £20 voucher for their time.

In Area 4, fieldwork was terminated early due to an underlying issue with this sample in which few claimants could be contacted (many claimants did not pick up the phone) and of those who did, few recalled the programme or had actually started the SWAP to which they had been referred. Only two interviews were completed from 207 recruitment calls, compared to 10 interviews completed from 88 recruitment calls in Area 3. The study team attempted to unpick the reasoning for the issues with the underlying sample in subsequent meetings and interviews with the local operational contacts, however, it was difficult to pinpoint this exactly. The information gathered suggested that the issue was likely a result of error(s) completing the local manual SWAP trackers. As a result, fieldwork was terminated early so that the findings could be reported to the timetable agreed.

Contacting employers and training providers

As described in the main report, the study was reliant on the case study areas to supply the contact details of employers and training providers who had taken part in a SWAP in their districts, as there was no alternative way of identifying these organisations. Within each area, the study aimed to interview a total of 7 employers, and 3 training providers, and so local contacts were asked to provide approximately 15 to 20 employer contacts and 5 to 10 training provider contacts to account for uncertainty in likely response rates. Obtaining contacts was more difficult in some of the case study areas and was affected by factors such as local record keeping of this information (for example, some training providers were listed as employers, and other contact information was out of date), and busyness of the staff involved. In all areas, subsequent samples of employers were requested due to poor response rates for this participant group.

To counter the risk of staff supplying only contacts for similar organisations, and therefore similar experiences of the programme, contact information for a range of organisations in terms of key characteristics (size, sector of SWAP , length of SWAP , number of SWAPs involved in, and how the SWAP was initiated) was requested. Organisations were then approached by researchers to ensure a spread across these characteristics, although achieving this was limited by response rates, particularly among employers.

Organisations were initially emailed using a template which explained the purpose of the research and asked if they were able to participate. Where organisations agreed to take part, they were then sent an additional information sheet and booked in for an interview at a convenient date and time. Where no response was received, a follow-up email was sent a few days later prompting them about the study. Finally, where the target number of interviews had not yet been reached, organisations were contacted by telephone for up to a maximum of two attempts. Where this was the case, the researcher verbally communicated the key information about the study contained in the initial emails.

For most areas, the first time employers and training providers heard about the research was when they were contacted via email about the study. In Area 4, however, DWP staff approached employers in advance before handing contact details over to the study team. This approach was taken as DWP staff in this area felt it would be beneficial in securing employer participation and minimised any risk to their relationships with these contacts if the study team were to contact them without warning. It should be noted that this may have increased the risk that some employers may have felt obligated to take part in the research and/or restricted their feedback due to a perceived lack of separation between DWP researchers and operational staff leading on SWAPs . As with all areas, researchers in Area 4 emphasised their independence from jobcentres ( JCPs ) and SWAP policy decision-making during each contact with participants, and the questions asked during data collection were framed in a way to encourage and enable participants to be honest about their experience. Despite this, it’s likely that a certain level of bias related to this aspect remains in the dataset obtained.

Each interview lasted approximately 30 minutes to an hour, depending on how much each organisation wanted to share. To ensure the most appropriate person was spoken to, the information shared during the recruitment stages requested that the participant was an individual who had knowledge of, or was responsible for, the SWAP that their organisation had been involved in.

Contacting staff

Once JCP Service Leaders had agreed for fieldwork to take place in their district areas, the study team were signposted to operational staff who would be able to facilitate the research. These individuals became key contacts for the study team during fieldwork. In initial meetings, these local contacts provided a broad overview of the SWAP set-up in their district, and the types of staff involved in delivery, from which a list of different staff roles to speak to as part of the fieldwork was agreed. Due to the varying nature of the local staffing models, it was easier to understand how SWAP delivery was organised in some areas more than others.

The project manager and case study leads maintained regular contact with these local contacts while fieldwork took place in each area. In Areas 1 and 2 this mostly consisted of contact via email, whereas for Areas 3 and 4 this took the form of a weekly scheduled meeting. In Areas 2, 3 and 4, a follow-up meeting took place with the local contacts to check the study team’s understanding of local SWAP delivery obtained through data collection, and to clarify any aspects of delivery that remained unclear.

The local contacts provided a list of suggested staff who could be approached for the fieldwork based on their role and involvement in local SWAP delivery. The study team then arranged the interviews and focus groups for these staff around their availability. In setting up the interviews and focus groups, an information sheet was provided about the research, and it was emphasised that their participation was voluntary. Despite this, some staff may have only participated in the study because they felt obligated to. As with other participant groups, the independence of the study team was emphasised, and participants were offered the opportunity to withdraw from the study if they wanted to.

Piloting [footnote 1] interviews were conducted with two members of DWP staff, one employer and one training provider who agreed to this. These interviews were conducted to test the length and appropriateness of the topic guides for these participant groups, and the quality of the data obtained. These individuals were recruited from a separate JCP district to the case study areas and so the information collected was not used in the analysis and reporting of this study. The topic guides were amended following these pilots.

For claimants, the first week of fieldwork was considered a pilot. Minimal changes were made to the topic guide following these interviews, and so, unlike the other fieldwork strands, the data collected during these interviews was analysed and reported on. It should, however, be noted that the topic guides and fieldwork processes were continually reviewed and modified throughout the data collection periods to ensure they were as efficient and effective as possible. The study team met multiple times a week to reflect on interviews, and formal debrief sessions were held within the study team, and separately with wider supporting researchers, following the end of data collection in each study area. This process allowed learning from each area to be implemented in subsequent fieldwork.

Analysis and Reporting

Once all interviews had been conducted, the interview notes formed the final dataset. The dataset was explored using a thematic analysis approach. As there were multiple researchers involved in the coding of the data, a coding framework (Annex 4) was developed to ensure consistency in coding across the study team. The research questions were used as a guide to ensure the framework aligned with the objectives of the research, and the framework was tested with an initial sample of interviews before a final version was agreed for coding the rest of the data (although this was still subject to ongoing tweaks as coding progressed).

Members of the study team were paired up to code a specific strand of data (for example, employers) and each pair coded the same initial set of notes to check alignment in their coding approach, before separately coding the remaining data. A separate member of the study team then examined a selection of coded notes from each pair to quality assure the completed coding. Feedback on the coding approach, particularly inconsistencies within each pair, was provided to the coders so that this could be incorporated into the analysis of future notes.

The project team met multiple times to discuss and agree the themes identified within the coded data. The themes identified via this process of analysis structured the findings within this report. When analysing the data, findings were explored by participant group (for example, claimants vs. employers) as well as by case study area (for example, Area 1 vs. Area 2), and these were included in the reporting where relevant.

A Quality Assurance ( QA ) panel was established to review the work of the research team during the analysis stage. The panel included researchers external to the project, senior researchers, and a fieldworker external to the study team, who was involved in conducting the research. This panel was engaged to review the initial coding framework that had been developed, and again to review how the codes had been applied to a sample of the data collected. This ensured that the approach taken to analysis had been peer reviewed, and that the data analysis conducted was of good quality. The final report was separately quality assured by an academic on secondment to the In-House Research Unit ( IHRU ), as well as senior researchers in the unit.

Annex 4: Initial coding framework

A pilot is a small-scale, preliminary study that is used as a test run for a particular research instrument to ensure its efficacy.  ↩

Is this page useful?

• Yes this page is useful
• No this page is not useful

Help us improve GOV.UK

Don’t include personal or financial information like your National Insurance number or credit card details.

To help us improve GOV.UK, we’d like to know more about your visit today. Please fill in this survey (opens in a new tab) .

ORIGINAL RESEARCH article

This article is part of the research topic.

Building the Future of Education Together: Innovation, Complexity, Sustainability, Interdisciplinary Research and Open Science

Developing the Skills for Complex Thinking Research: A Case Study Using Social Robotics to Produce Scientific Papers Provisionally Accepted

• 1 Institute for the Future of Education, Monterrey Institute of Technology and Higher Education (ITESM), Mexico
• 2 University of Cienfuegos, Cuba

The final, formatted version of the article will be published soon.

The development of university students' skills to successfully produce scientific documents has been a recurring topic of study in academia. This paper analyzes the implementation of a training experience using a digital environment mediated by video content materials starring humanoid robots. The research aimed to scale complex thinking and its subcompetencies as a hinge to strengthen basic academic research skills. Students from Colombia, Ecuador, and Mexico committed to preparing a scientific document as part of their professional training participated. A pretest to know their initial level of perception, a posttest to evaluate if there was a change, and a scientific document the students delivered at the end of the training experience comprised the methodology to demonstrate the improvement of their skills. The results indicated students' perceived improvement in the sub-competencies of systemic, creative, scientific, and innovative thinking; however, their perceptions did not align with that of the tutor who reviewed the delivered scientific product. The conclusion was that although the training experience helped strengthen the students' skills, variables that are determinants for a student to develop the knowledge necessary to prepare scientific documents and their derived products remain to be analyzed.

Keywords: higher education, research skills, Educational innovation, complex thinking, scientific thinking, Critical Thinking, Innovative thinking, social robotics

Received: 16 Oct 2023; Accepted: 17 May 2024.

Copyright: © 2024 Lopez-Caudana, George-Reyes and Avello-Martínez. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Dr. Edgar O. Lopez-Caudana, Institute for the Future of Education, Monterrey Institute of Technology and Higher Education (ITESM), Monterrey, Mexico

People also looked at