literature based research methodology

Chapter Four: Theory, Methodologies, Methods, and Evidence

Research Methods

You are viewing the first edition of this textbook. a second edition is available – please visit the latest edition for updated information..

This page discusses the following topics:

Research Goals

Research method types.

Before discussing research   methods , we need to distinguish them from  methodologies  and  research skills . Methodologies, linked to literary theories, are tools and lines of investigation: sets of practices and propositions about texts and the world. Researchers using Marxist literary criticism will adopt methodologies that look to material forces like labor, ownership, and technology to understand literature and its relationship to the world. They will also seek to understand authors not as inspired geniuses but as people whose lives and work are shaped by social forces.

Example: Critical Race Theory Methodologies

Critical Race Theory may use a variety of methodologies, including

• Interest convergence: investigating whether marginalized groups only achieve progress when dominant groups benefit as well
• Intersectional theory: investigating how multiple factors of advantage and disadvantage around race, gender, ethnicity, religion, etc. operate together in complex ways
• Radical critique of the law: investigating how the law has historically been used to marginalize particular groups, such as black people, while recognizing that legal efforts are important to achieve emancipation and civil rights
• Social constructivism: investigating how race is socially constructed (rather than biologically grounded)
• Standpoint epistemology: investigating how knowledge relates to social position
• Structural determinism: investigating how structures of thought and of organizations determine social outcomes

To identify appropriate methodologies, you will need to research your chosen theory and gather what methodologies are associated with it. For the most part, we can’t assume that there are “one size fits all” methodologies.

Research skills are about how you handle materials such as library search engines, citation management programs, special collections materials, and so on.

Research methods  are about where and how you get answers to your research questions. Are you conducting interviews? Visiting archives? Doing close readings? Reviewing scholarship? You will need to choose which methods are most appropriate to use in your research and you need to gain some knowledge about how to use these methods. In other words, you need to do some research into research methods!

Your choice of research method depends on the kind of questions you are asking. For example, if you want to understand how an author progressed through several drafts to arrive at a final manuscript, you may need to do archival research. If you want to understand why a particular literary work became a bestseller, you may need to do audience research. If you want to know why a contemporary author wrote a particular work, you may need to do interviews. Usually literary research involves a combination of methods such as  archival research ,  discourse analysis , and  qualitative research  methods.

Literary research methods tend to differ from research methods in the hard sciences (such as physics and chemistry). Science research must present results that are reproducible, while literary research rarely does (though it must still present evidence for its claims). Literary research often deals with questions of meaning, social conventions, representations of lived experience, and aesthetic effects; these are questions that reward dialogue and different perspectives rather than one great experiment that settles the issue. In literary research, we might get many valuable answers even though they are quite different from one another. Also in literary research, we usually have some room to speculate about answers, but our claims have to be plausible (believable) and our argument comprehensive (meaning we don’t overlook evidence that would alter our argument significantly if it were known).

A literary researcher might select the following:

Theory: Critical Race Theory

Methodology: Social Constructivism

Method: Scholarly

Skills: Search engines, citation management

Wendy Belcher, in  Writing Your Journal Article in 12 Weeks , identifies two main approaches to understanding literary works: looking at a text by itself (associated with New Criticism ) and looking at texts as they connect to society (associated with Cultural Studies ). The goal of New Criticism is to bring the reader further into the text. The goal of Cultural Studies is to bring the reader into the network of discourses that surround and pass through the text. Other approaches, such as Ecocriticism, relate literary texts to the Sciences (as well as to the Humanities).

The New Critics, starting in the 1940s,  focused on meaning within the text itself, using a method they called “ close reading .” The text itself becomes e vidence for a particular reading. Using this approach, you should summarize the literary work briefly and q uote particularly meaningful passages, being sure to introduce quotes and then interpret them (never let them stand alone). Make connections within the work; a sk  “why” and “how” the various parts of the text relate to each other.

Cultural Studies critics see all texts  as connected to society; the critic  therefore has to connect a text to at least one political or social issue. How and why does  the text reproduce particular knowledge systems (known as discourses) and how do these knowledge systems relate to issues of power within the society? Who speaks and when? Answering these questions helps your reader understand the text in context. Cultural contexts can include the treatment of gender (Feminist, Queer), class (Marxist), nationality, race, religion, or any other area of human society.

Other approaches, such as psychoanalytic literary criticism , look at literary texts to better understand human psychology. A psychoanalytic reading can focus on a character, the author, the reader, or on society in general. Ecocriticism  look at human understandings of nature in literary texts.

We select our research methods based on the kinds of things we want to know. For example, we may be studying the relationship between literature and society, between author and text, or the status of a work in the literary canon. We may want to know about a work’s form, genre, or thematics. We may want to know about the audience’s reading and reception, or about methods for teaching literature in schools.

Below are a few research methods and their descriptions. You may need to consult with your instructor about which ones are most appropriate for your project. The first list covers methods most students use in their work. The second list covers methods more commonly used by advanced researchers. Even if you will not be using methods from this second list in your research project, you may read about these research methods in the scholarship you find.

Most commonly used undergraduate research methods:

• Scholarship Methods:  Studies the body of scholarship written about a particular author, literary work, historical period, literary movement, genre, theme, theory, or method.
• Textual Analysis Methods:  Used for close readings of literary texts, these methods also rely on literary theory and background information to support the reading.
• Biographical Methods:  Used to study the life of the author to better understand their work and times, these methods involve reading biographies and autobiographies about the author, and may also include research into private papers, correspondence, and interviews.
• Discourse Analysis Methods:  Studies language patterns to reveal ideology and social relations of power. This research involves the study of institutions, social groups, and social movements to understand how people in various settings use language to represent the world to themselves and others. Literary works may present complex mixtures of discourses which the characters (and readers) have to navigate.
• Creative Writing Methods:  A literary re-working of another literary text, creative writing research is used to better understand a literary work by investigating its language, formal structures, composition methods, themes, and so on. For instance, a creative research project may retell a story from a minor character’s perspective to reveal an alternative reading of events. To qualify as research, a creative research project is usually combined with a piece of theoretical writing that explains and justifies the work.

Methods used more often by advanced researchers:

• Archival Methods: Usually involves trips to special collections where original papers are kept. In these archives are many unpublished materials such as diaries, letters, photographs, ledgers, and so on. These materials can offer us invaluable insight into the life of an author, the development of a literary work, or the society in which the author lived. There are at least three major archives of James Baldwin’s papers: The Smithsonian , Yale , and The New York Public Library . Descriptions of such materials are often available online, but the materials themselves are typically stored in boxes at the archive.
• Computational Methods:  Used for statistical analysis of texts such as studies of the popularity and meaning of particular words in literature over time.
• Ethnographic Methods:  Studies groups of people and their interactions with literary works, for instance in educational institutions, in reading groups (such as book clubs), and in fan networks. This approach may involve interviews and visits to places (including online communities) where people interact with literary works. Note: before you begin such work, you must have  Institutional Review Board (IRB)  approval “to protect the rights and welfare of human participants involved in research.”
• Visual Methods:  Studies the visual qualities of literary works. Some literary works, such as illuminated manuscripts, children’s literature, and graphic novels, present a complex interplay of text and image. Even works without illustrations can be studied for their use of typography, layout, and other visual features.

Regardless of the method(s) you choose, you will need to learn how to apply them to your work and how to carry them out successfully. For example, you should know that many archives do not allow you to bring pens (you can use pencils) and you may not be allowed to bring bags into the archives. You will need to keep a record of which documents you consult and their location (box number, etc.) in the archives. If you are unsure how to use a particular method, please consult a book about it. [1] Also, ask for the advice of trained researchers such as your instructor or a research librarian.

• What research method(s) will you be using for your paper? Why did you make this method selection over other methods? If you haven’t made a selection yet, which methods are you considering?
• What specific methodological approaches are you most interested in exploring in relation to the chosen literary work?
• What is your plan for researching your method(s) and its major approaches?
• What was the most important lesson you learned from this page? What point was confusing or difficult to understand?

Write your answers in a webcourse discussion page.

• Introduction to Research Methods: A Practical Guide for Anyone Undertaking a Research Project  by Catherine, Dr. Dawson
• Practical Research Methods: A User-Friendly Guide to Mastering Research Techniques and Projects  by Catherine Dawson
• Qualitative Inquiry and Research Design: Choosing Among Five Approaches  by John W. Creswell  Cheryl N. Poth
• Qualitative Research Evaluation Methods: Integrating Theory and Practice  by Michael Quinn Patton
• Research Design: Qualitative, Quantitative, and Mixed Methods Approaches  by John W. Creswell  J. David Creswell
• Research Methodology: A Step-by-Step Guide for Beginners  by Ranjit Kumar
• Research Methodology: Methods and Techniques  by C.R. Kothari

Strategies for Conducting Literary Research Copyright © 2021 by Barry Mauer & John Venecek is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Research Methods

• Getting Started
• Literature Review Research
• Research Design
• Research Design By Discipline
• SAGE Research Methods
• Teaching with SAGE Research Methods

Literature Review

• What is a Literature Review?
• What is NOT a Literature Review?
• Purposes of a Literature Review
• Types of Literature Reviews
• Literature Reviews vs. Systematic Reviews
• Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

• Summarizes and analyzes previous research relevant to a topic
• Includes scholarly books and articles published in academic journals
• Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

• Help gather ideas or information
• Keep up to date in current trends and findings
• Help develop new questions

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

• provides thorough knowledge of previous studies; introduces seminal works.
• helps focus one’s own research topic.
• identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
• suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
• identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
• helps the researcher avoid repetition of earlier research.
• suggests unexplored populations.
• determines whether past studies agree or disagree; identifies controversy in the literature.
• tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

• clearly stated objectives with pre-defined eligibility criteria for studies
• explicit, reproducible methodology
• a systematic search that attempts to identify all studies
• assessment of the validity of the findings of the included studies (e.g. risk of bias)
• systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

• << Previous: Getting Started
• Next: Research Design >>
• Last Updated: Aug 21, 2023 4:07 PM
• URL: https://guides.lib.udel.edu/researchmethods

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: May 15, 2024 9:53 AM
• URL: https://libguides.usc.edu/writingguide

• Open access
• Published: 07 September 2020

A tutorial on methodological studies: the what, when, how and why

• Lawrence Mbuagbaw   ORCID: orcid.org/0000-0001-5855-5461 1 , 2 , 3 ,
• Daeria O. Lawson 1 ,
• Livia Puljak 4 ,
• David B. Allison 5 &
• Lehana Thabane 1 , 2 , 6 , 7 , 8  

BMC Medical Research Methodology volume  20 , Article number:  226 ( 2020 ) Cite this article

39k Accesses

53 Citations

58 Altmetric

Metrics details

Methodological studies – studies that evaluate the design, analysis or reporting of other research-related reports – play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a “frequently asked questions” format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

Peer Review reports

The field of meta-research (or research-on-research) has proliferated in recent years in response to issues with research quality and conduct [ 1 , 2 , 3 ]. As the name suggests, this field targets issues with research design, conduct, analysis and reporting. Various types of research reports are often examined as the unit of analysis in these studies (e.g. abstracts, full manuscripts, trial registry entries). Like many other novel fields of research, meta-research has seen a proliferation of use before the development of reporting guidance. For example, this was the case with randomized trials for which risk of bias tools and reporting guidelines were only developed much later – after many trials had been published and noted to have limitations [ 4 , 5 ]; and for systematic reviews as well [ 6 , 7 , 8 ]. However, in the absence of formal guidance, studies that report on research differ substantially in how they are named, conducted and reported [ 9 , 10 ]. This creates challenges in identifying, summarizing and comparing them. In this tutorial paper, we will use the term methodological study to refer to any study that reports on the design, conduct, analysis or reporting of primary or secondary research-related reports (such as trial registry entries and conference abstracts).

In the past 10 years, there has been an increase in the use of terms related to methodological studies (based on records retrieved with a keyword search [in the title and abstract] for “methodological review” and “meta-epidemiological study” in PubMed up to December 2019), suggesting that these studies may be appearing more frequently in the literature. See Fig.  1 .

Trends in the number studies that mention “methodological review” or “meta-

epidemiological study” in PubMed.

The methods used in many methodological studies have been borrowed from systematic and scoping reviews. This practice has influenced the direction of the field, with many methodological studies including searches of electronic databases, screening of records, duplicate data extraction and assessments of risk of bias in the included studies. However, the research questions posed in methodological studies do not always require the approaches listed above, and guidance is needed on when and how to apply these methods to a methodological study. Even though methodological studies can be conducted on qualitative or mixed methods research, this paper focuses on and draws examples exclusively from quantitative research.

The objectives of this paper are to provide some insights on how to conduct methodological studies so that there is greater consistency between the research questions posed, and the design, analysis and reporting of findings. We provide multiple examples to illustrate concepts and a proposed framework for categorizing methodological studies in quantitative research.

What is a methodological study?

Any study that describes or analyzes methods (design, conduct, analysis or reporting) in published (or unpublished) literature is a methodological study. Consequently, the scope of methodological studies is quite extensive and includes, but is not limited to, topics as diverse as: research question formulation [ 11 ]; adherence to reporting guidelines [ 12 , 13 , 14 ] and consistency in reporting [ 15 ]; approaches to study analysis [ 16 ]; investigating the credibility of analyses [ 17 ]; and studies that synthesize these methodological studies [ 18 ]. While the nomenclature of methodological studies is not uniform, the intents and purposes of these studies remain fairly consistent – to describe or analyze methods in primary or secondary studies. As such, methodological studies may also be classified as a subtype of observational studies.

Parallel to this are experimental studies that compare different methods. Even though they play an important role in informing optimal research methods, experimental methodological studies are beyond the scope of this paper. Examples of such studies include the randomized trials by Buscemi et al., comparing single data extraction to double data extraction [ 19 ], and Carrasco-Labra et al., comparing approaches to presenting findings in Grading of Recommendations, Assessment, Development and Evaluations (GRADE) summary of findings tables [ 20 ]. In these studies, the unit of analysis is the person or groups of individuals applying the methods. We also direct readers to the Studies Within a Trial (SWAT) and Studies Within a Review (SWAR) programme operated through the Hub for Trials Methodology Research, for further reading as a potential useful resource for these types of experimental studies [ 21 ]. Lastly, this paper is not meant to inform the conduct of research using computational simulation and mathematical modeling for which some guidance already exists [ 22 ], or studies on the development of methods using consensus-based approaches.

When should we conduct a methodological study?

Methodological studies occupy a unique niche in health research that allows them to inform methodological advances. Methodological studies should also be conducted as pre-cursors to reporting guideline development, as they provide an opportunity to understand current practices, and help to identify the need for guidance and gaps in methodological or reporting quality. For example, the development of the popular Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) guidelines were preceded by methodological studies identifying poor reporting practices [ 23 , 24 ]. In these instances, after the reporting guidelines are published, methodological studies can also be used to monitor uptake of the guidelines.

These studies can also be conducted to inform the state of the art for design, analysis and reporting practices across different types of health research fields, with the aim of improving research practices, and preventing or reducing research waste. For example, Samaan et al. conducted a scoping review of adherence to different reporting guidelines in health care literature [ 18 ]. Methodological studies can also be used to determine the factors associated with reporting practices. For example, Abbade et al. investigated journal characteristics associated with the use of the Participants, Intervention, Comparison, Outcome, Timeframe (PICOT) format in framing research questions in trials of venous ulcer disease [ 11 ].

How often are methodological studies conducted?

There is no clear answer to this question. Based on a search of PubMed, the use of related terms (“methodological review” and “meta-epidemiological study”) – and therefore, the number of methodological studies – is on the rise. However, many other terms are used to describe methodological studies. There are also many studies that explore design, conduct, analysis or reporting of research reports, but that do not use any specific terms to describe or label their study design in terms of “methodology”. This diversity in nomenclature makes a census of methodological studies elusive. Appropriate terminology and key words for methodological studies are needed to facilitate improved accessibility for end-users.

Why do we conduct methodological studies?

Methodological studies provide information on the design, conduct, analysis or reporting of primary and secondary research and can be used to appraise quality, quantity, completeness, accuracy and consistency of health research. These issues can be explored in specific fields, journals, databases, geographical regions and time periods. For example, Areia et al. explored the quality of reporting of endoscopic diagnostic studies in gastroenterology [ 25 ]; Knol et al. investigated the reporting of p -values in baseline tables in randomized trial published in high impact journals [ 26 ]; Chen et al. describe adherence to the Consolidated Standards of Reporting Trials (CONSORT) statement in Chinese Journals [ 27 ]; and Hopewell et al. describe the effect of editors’ implementation of CONSORT guidelines on reporting of abstracts over time [ 28 ]. Methodological studies provide useful information to researchers, clinicians, editors, publishers and users of health literature. As a result, these studies have been at the cornerstone of important methodological developments in the past two decades and have informed the development of many health research guidelines including the highly cited CONSORT statement [ 5 ].

Where can we find methodological studies?

Methodological studies can be found in most common biomedical bibliographic databases (e.g. Embase, MEDLINE, PubMed, Web of Science). However, the biggest caveat is that methodological studies are hard to identify in the literature due to the wide variety of names used and the lack of comprehensive databases dedicated to them. A handful can be found in the Cochrane Library as “Cochrane Methodology Reviews”, but these studies only cover methodological issues related to systematic reviews. Previous attempts to catalogue all empirical studies of methods used in reviews were abandoned 10 years ago [ 29 ]. In other databases, a variety of search terms may be applied with different levels of sensitivity and specificity.

Some frequently asked questions about methodological studies

In this section, we have outlined responses to questions that might help inform the conduct of methodological studies.

Q: How should I select research reports for my methodological study?

A: Selection of research reports for a methodological study depends on the research question and eligibility criteria. Once a clear research question is set and the nature of literature one desires to review is known, one can then begin the selection process. Selection may begin with a broad search, especially if the eligibility criteria are not apparent. For example, a methodological study of Cochrane Reviews of HIV would not require a complex search as all eligible studies can easily be retrieved from the Cochrane Library after checking a few boxes [ 30 ]. On the other hand, a methodological study of subgroup analyses in trials of gastrointestinal oncology would require a search to find such trials, and further screening to identify trials that conducted a subgroup analysis [ 31 ].

The strategies used for identifying participants in observational studies can apply here. One may use a systematic search to identify all eligible studies. If the number of eligible studies is unmanageable, a random sample of articles can be expected to provide comparable results if it is sufficiently large [ 32 ]. For example, Wilson et al. used a random sample of trials from the Cochrane Stroke Group’s Trial Register to investigate completeness of reporting [ 33 ]. It is possible that a simple random sample would lead to underrepresentation of units (i.e. research reports) that are smaller in number. This is relevant if the investigators wish to compare multiple groups but have too few units in one group. In this case a stratified sample would help to create equal groups. For example, in a methodological study comparing Cochrane and non-Cochrane reviews, Kahale et al. drew random samples from both groups [ 34 ]. Alternatively, systematic or purposeful sampling strategies can be used and we encourage researchers to justify their selected approaches based on the study objective.

Q: How many databases should I search?

A: The number of databases one should search would depend on the approach to sampling, which can include targeting the entire “population” of interest or a sample of that population. If you are interested in including the entire target population for your research question, or drawing a random or systematic sample from it, then a comprehensive and exhaustive search for relevant articles is required. In this case, we recommend using systematic approaches for searching electronic databases (i.e. at least 2 databases with a replicable and time stamped search strategy). The results of your search will constitute a sampling frame from which eligible studies can be drawn.

Alternatively, if your approach to sampling is purposeful, then we recommend targeting the database(s) or data sources (e.g. journals, registries) that include the information you need. For example, if you are conducting a methodological study of high impact journals in plastic surgery and they are all indexed in PubMed, you likely do not need to search any other databases. You may also have a comprehensive list of all journals of interest and can approach your search using the journal names in your database search (or by accessing the journal archives directly from the journal’s website). Even though one could also search journals’ web pages directly, using a database such as PubMed has multiple advantages, such as the use of filters, so the search can be narrowed down to a certain period, or study types of interest. Furthermore, individual journals’ web sites may have different search functionalities, which do not necessarily yield a consistent output.

Q: Should I publish a protocol for my methodological study?

A: A protocol is a description of intended research methods. Currently, only protocols for clinical trials require registration [ 35 ]. Protocols for systematic reviews are encouraged but no formal recommendation exists. The scientific community welcomes the publication of protocols because they help protect against selective outcome reporting, the use of post hoc methodologies to embellish results, and to help avoid duplication of efforts [ 36 ]. While the latter two risks exist in methodological research, the negative consequences may be substantially less than for clinical outcomes. In a sample of 31 methodological studies, 7 (22.6%) referenced a published protocol [ 9 ]. In the Cochrane Library, there are 15 protocols for methodological reviews (21 July 2020). This suggests that publishing protocols for methodological studies is not uncommon.

Authors can consider publishing their study protocol in a scholarly journal as a manuscript. Advantages of such publication include obtaining peer-review feedback about the planned study, and easy retrieval by searching databases such as PubMed. The disadvantages in trying to publish protocols includes delays associated with manuscript handling and peer review, as well as costs, as few journals publish study protocols, and those journals mostly charge article-processing fees [ 37 ]. Authors who would like to make their protocol publicly available without publishing it in scholarly journals, could deposit their study protocols in publicly available repositories, such as the Open Science Framework ( https://osf.io/ ).

Q: How to appraise the quality of a methodological study?

A: To date, there is no published tool for appraising the risk of bias in a methodological study, but in principle, a methodological study could be considered as a type of observational study. Therefore, during conduct or appraisal, care should be taken to avoid the biases common in observational studies [ 38 ]. These biases include selection bias, comparability of groups, and ascertainment of exposure or outcome. In other words, to generate a representative sample, a comprehensive reproducible search may be necessary to build a sampling frame. Additionally, random sampling may be necessary to ensure that all the included research reports have the same probability of being selected, and the screening and selection processes should be transparent and reproducible. To ensure that the groups compared are similar in all characteristics, matching, random sampling or stratified sampling can be used. Statistical adjustments for between-group differences can also be applied at the analysis stage. Finally, duplicate data extraction can reduce errors in assessment of exposures or outcomes.

Q: Should I justify a sample size?

A: In all instances where one is not using the target population (i.e. the group to which inferences from the research report are directed) [ 39 ], a sample size justification is good practice. The sample size justification may take the form of a description of what is expected to be achieved with the number of articles selected, or a formal sample size estimation that outlines the number of articles required to answer the research question with a certain precision and power. Sample size justifications in methodological studies are reasonable in the following instances:

Comparing two groups

Determining a proportion, mean or another quantifier

Determining factors associated with an outcome using regression-based analyses

For example, El Dib et al. computed a sample size requirement for a methodological study of diagnostic strategies in randomized trials, based on a confidence interval approach [ 40 ].

Q: What should I call my study?

A: Other terms which have been used to describe/label methodological studies include “ methodological review ”, “methodological survey” , “meta-epidemiological study” , “systematic review” , “systematic survey”, “meta-research”, “research-on-research” and many others. We recommend that the study nomenclature be clear, unambiguous, informative and allow for appropriate indexing. Methodological study nomenclature that should be avoided includes “ systematic review” – as this will likely be confused with a systematic review of a clinical question. “ Systematic survey” may also lead to confusion about whether the survey was systematic (i.e. using a preplanned methodology) or a survey using “ systematic” sampling (i.e. a sampling approach using specific intervals to determine who is selected) [ 32 ]. Any of the above meanings of the words “ systematic” may be true for methodological studies and could be potentially misleading. “ Meta-epidemiological study” is ideal for indexing, but not very informative as it describes an entire field. The term “ review ” may point towards an appraisal or “review” of the design, conduct, analysis or reporting (or methodological components) of the targeted research reports, yet it has also been used to describe narrative reviews [ 41 , 42 ]. The term “ survey ” is also in line with the approaches used in many methodological studies [ 9 ], and would be indicative of the sampling procedures of this study design. However, in the absence of guidelines on nomenclature, the term “ methodological study ” is broad enough to capture most of the scenarios of such studies.

Q: Should I account for clustering in my methodological study?

A: Data from methodological studies are often clustered. For example, articles coming from a specific source may have different reporting standards (e.g. the Cochrane Library). Articles within the same journal may be similar due to editorial practices and policies, reporting requirements and endorsement of guidelines. There is emerging evidence that these are real concerns that should be accounted for in analyses [ 43 ]. Some cluster variables are described in the section: “ What variables are relevant to methodological studies?”

A variety of modelling approaches can be used to account for correlated data, including the use of marginal, fixed or mixed effects regression models with appropriate computation of standard errors [ 44 ]. For example, Kosa et al. used generalized estimation equations to account for correlation of articles within journals [ 15 ]. Not accounting for clustering could lead to incorrect p -values, unduly narrow confidence intervals, and biased estimates [ 45 ].

Q: Should I extract data in duplicate?

A: Yes. Duplicate data extraction takes more time but results in less errors [ 19 ]. Data extraction errors in turn affect the effect estimate [ 46 ], and therefore should be mitigated. Duplicate data extraction should be considered in the absence of other approaches to minimize extraction errors. However, much like systematic reviews, this area will likely see rapid new advances with machine learning and natural language processing technologies to support researchers with screening and data extraction [ 47 , 48 ]. However, experience plays an important role in the quality of extracted data and inexperienced extractors should be paired with experienced extractors [ 46 , 49 ].

Q: Should I assess the risk of bias of research reports included in my methodological study?

A : Risk of bias is most useful in determining the certainty that can be placed in the effect measure from a study. In methodological studies, risk of bias may not serve the purpose of determining the trustworthiness of results, as effect measures are often not the primary goal of methodological studies. Determining risk of bias in methodological studies is likely a practice borrowed from systematic review methodology, but whose intrinsic value is not obvious in methodological studies. When it is part of the research question, investigators often focus on one aspect of risk of bias. For example, Speich investigated how blinding was reported in surgical trials [ 50 ], and Abraha et al., investigated the application of intention-to-treat analyses in systematic reviews and trials [ 51 ].

Q: What variables are relevant to methodological studies?

A: There is empirical evidence that certain variables may inform the findings in a methodological study. We outline some of these and provide a brief overview below:

Country: Countries and regions differ in their research cultures, and the resources available to conduct research. Therefore, it is reasonable to believe that there may be differences in methodological features across countries. Methodological studies have reported loco-regional differences in reporting quality [ 52 , 53 ]. This may also be related to challenges non-English speakers face in publishing papers in English.

Authors’ expertise: The inclusion of authors with expertise in research methodology, biostatistics, and scientific writing is likely to influence the end-product. Oltean et al. found that among randomized trials in orthopaedic surgery, the use of analyses that accounted for clustering was more likely when specialists (e.g. statistician, epidemiologist or clinical trials methodologist) were included on the study team [ 54 ]. Fleming et al. found that including methodologists in the review team was associated with appropriate use of reporting guidelines [ 55 ].

Source of funding and conflicts of interest: Some studies have found that funded studies report better [ 56 , 57 ], while others do not [ 53 , 58 ]. The presence of funding would indicate the availability of resources deployed to ensure optimal design, conduct, analysis and reporting. However, the source of funding may introduce conflicts of interest and warrant assessment. For example, Kaiser et al. investigated the effect of industry funding on obesity or nutrition randomized trials and found that reporting quality was similar [ 59 ]. Thomas et al. looked at reporting quality of long-term weight loss trials and found that industry funded studies were better [ 60 ]. Kan et al. examined the association between industry funding and “positive trials” (trials reporting a significant intervention effect) and found that industry funding was highly predictive of a positive trial [ 61 ]. This finding is similar to that of a recent Cochrane Methodology Review by Hansen et al. [ 62 ]

Journal characteristics: Certain journals’ characteristics may influence the study design, analysis or reporting. Characteristics such as journal endorsement of guidelines [ 63 , 64 ], and Journal Impact Factor (JIF) have been shown to be associated with reporting [ 63 , 65 , 66 , 67 ].

Study size (sample size/number of sites): Some studies have shown that reporting is better in larger studies [ 53 , 56 , 58 ].

Year of publication: It is reasonable to assume that design, conduct, analysis and reporting of research will change over time. Many studies have demonstrated improvements in reporting over time or after the publication of reporting guidelines [ 68 , 69 ].

Type of intervention: In a methodological study of reporting quality of weight loss intervention studies, Thabane et al. found that trials of pharmacologic interventions were reported better than trials of non-pharmacologic interventions [ 70 ].

Interactions between variables: Complex interactions between the previously listed variables are possible. High income countries with more resources may be more likely to conduct larger studies and incorporate a variety of experts. Authors in certain countries may prefer certain journals, and journal endorsement of guidelines and editorial policies may change over time.

Q: Should I focus only on high impact journals?

A: Investigators may choose to investigate only high impact journals because they are more likely to influence practice and policy, or because they assume that methodological standards would be higher. However, the JIF may severely limit the scope of articles included and may skew the sample towards articles with positive findings. The generalizability and applicability of findings from a handful of journals must be examined carefully, especially since the JIF varies over time. Even among journals that are all “high impact”, variations exist in methodological standards.

Q: Can I conduct a methodological study of qualitative research?

A: Yes. Even though a lot of methodological research has been conducted in the quantitative research field, methodological studies of qualitative studies are feasible. Certain databases that catalogue qualitative research including the Cumulative Index to Nursing & Allied Health Literature (CINAHL) have defined subject headings that are specific to methodological research (e.g. “research methodology”). Alternatively, one could also conduct a qualitative methodological review; that is, use qualitative approaches to synthesize methodological issues in qualitative studies.

Q: What reporting guidelines should I use for my methodological study?

A: There is no guideline that covers the entire scope of methodological studies. One adaptation of the PRISMA guidelines has been published, which works well for studies that aim to use the entire target population of research reports [ 71 ]. However, it is not widely used (40 citations in 2 years as of 09 December 2019), and methodological studies that are designed as cross-sectional or before-after studies require a more fit-for purpose guideline. A more encompassing reporting guideline for a broad range of methodological studies is currently under development [ 72 ]. However, in the absence of formal guidance, the requirements for scientific reporting should be respected, and authors of methodological studies should focus on transparency and reproducibility.

Q: What are the potential threats to validity and how can I avoid them?

A: Methodological studies may be compromised by a lack of internal or external validity. The main threats to internal validity in methodological studies are selection and confounding bias. Investigators must ensure that the methods used to select articles does not make them differ systematically from the set of articles to which they would like to make inferences. For example, attempting to make extrapolations to all journals after analyzing high-impact journals would be misleading.

Many factors (confounders) may distort the association between the exposure and outcome if the included research reports differ with respect to these factors [ 73 ]. For example, when examining the association between source of funding and completeness of reporting, it may be necessary to account for journals that endorse the guidelines. Confounding bias can be addressed by restriction, matching and statistical adjustment [ 73 ]. Restriction appears to be the method of choice for many investigators who choose to include only high impact journals or articles in a specific field. For example, Knol et al. examined the reporting of p -values in baseline tables of high impact journals [ 26 ]. Matching is also sometimes used. In the methodological study of non-randomized interventional studies of elective ventral hernia repair, Parker et al. matched prospective studies with retrospective studies and compared reporting standards [ 74 ]. Some other methodological studies use statistical adjustments. For example, Zhang et al. used regression techniques to determine the factors associated with missing participant data in trials [ 16 ].

With regard to external validity, researchers interested in conducting methodological studies must consider how generalizable or applicable their findings are. This should tie in closely with the research question and should be explicit. For example. Findings from methodological studies on trials published in high impact cardiology journals cannot be assumed to be applicable to trials in other fields. However, investigators must ensure that their sample truly represents the target sample either by a) conducting a comprehensive and exhaustive search, or b) using an appropriate and justified, randomly selected sample of research reports.

Even applicability to high impact journals may vary based on the investigators’ definition, and over time. For example, for high impact journals in the field of general medicine, Bouwmeester et al. included the Annals of Internal Medicine (AIM), BMJ, the Journal of the American Medical Association (JAMA), Lancet, the New England Journal of Medicine (NEJM), and PLoS Medicine ( n  = 6) [ 75 ]. In contrast, the high impact journals selected in the methodological study by Schiller et al. were BMJ, JAMA, Lancet, and NEJM ( n  = 4) [ 76 ]. Another methodological study by Kosa et al. included AIM, BMJ, JAMA, Lancet and NEJM ( n  = 5). In the methodological study by Thabut et al., journals with a JIF greater than 5 were considered to be high impact. Riado Minguez et al. used first quartile journals in the Journal Citation Reports (JCR) for a specific year to determine “high impact” [ 77 ]. Ultimately, the definition of high impact will be based on the number of journals the investigators are willing to include, the year of impact and the JIF cut-off [ 78 ]. We acknowledge that the term “generalizability” may apply differently for methodological studies, especially when in many instances it is possible to include the entire target population in the sample studied.

Finally, methodological studies are not exempt from information bias which may stem from discrepancies in the included research reports [ 79 ], errors in data extraction, or inappropriate interpretation of the information extracted. Likewise, publication bias may also be a concern in methodological studies, but such concepts have not yet been explored.

A proposed framework

In order to inform discussions about methodological studies, the development of guidance for what should be reported, we have outlined some key features of methodological studies that can be used to classify them. For each of the categories outlined below, we provide an example. In our experience, the choice of approach to completing a methodological study can be informed by asking the following four questions:

What is the aim?

Methodological studies that investigate bias

A methodological study may be focused on exploring sources of bias in primary or secondary studies (meta-bias), or how bias is analyzed. We have taken care to distinguish bias (i.e. systematic deviations from the truth irrespective of the source) from reporting quality or completeness (i.e. not adhering to a specific reporting guideline or norm). An example of where this distinction would be important is in the case of a randomized trial with no blinding. This study (depending on the nature of the intervention) would be at risk of performance bias. However, if the authors report that their study was not blinded, they would have reported adequately. In fact, some methodological studies attempt to capture both “quality of conduct” and “quality of reporting”, such as Richie et al., who reported on the risk of bias in randomized trials of pharmacy practice interventions [ 80 ]. Babic et al. investigated how risk of bias was used to inform sensitivity analyses in Cochrane reviews [ 81 ]. Further, biases related to choice of outcomes can also be explored. For example, Tan et al investigated differences in treatment effect size based on the outcome reported [ 82 ].

Methodological studies that investigate quality (or completeness) of reporting

Methodological studies may report quality of reporting against a reporting checklist (i.e. adherence to guidelines) or against expected norms. For example, Croituro et al. report on the quality of reporting in systematic reviews published in dermatology journals based on their adherence to the PRISMA statement [ 83 ], and Khan et al. described the quality of reporting of harms in randomized controlled trials published in high impact cardiovascular journals based on the CONSORT extension for harms [ 84 ]. Other methodological studies investigate reporting of certain features of interest that may not be part of formally published checklists or guidelines. For example, Mbuagbaw et al. described how often the implications for research are elaborated using the Evidence, Participants, Intervention, Comparison, Outcome, Timeframe (EPICOT) format [ 30 ].

Methodological studies that investigate the consistency of reporting

Sometimes investigators may be interested in how consistent reports of the same research are, as it is expected that there should be consistency between: conference abstracts and published manuscripts; manuscript abstracts and manuscript main text; and trial registration and published manuscript. For example, Rosmarakis et al. investigated consistency between conference abstracts and full text manuscripts [ 85 ].

Methodological studies that investigate factors associated with reporting

In addition to identifying issues with reporting in primary and secondary studies, authors of methodological studies may be interested in determining the factors that are associated with certain reporting practices. Many methodological studies incorporate this, albeit as a secondary outcome. For example, Farrokhyar et al. investigated the factors associated with reporting quality in randomized trials of coronary artery bypass grafting surgery [ 53 ].

Methodological studies that investigate methods

Methodological studies may also be used to describe methods or compare methods, and the factors associated with methods. Muller et al. described the methods used for systematic reviews and meta-analyses of observational studies [ 86 ].

Methodological studies that summarize other methodological studies

Some methodological studies synthesize results from other methodological studies. For example, Li et al. conducted a scoping review of methodological reviews that investigated consistency between full text and abstracts in primary biomedical research [ 87 ].

Methodological studies that investigate nomenclature and terminology

Some methodological studies may investigate the use of names and terms in health research. For example, Martinic et al. investigated the definitions of systematic reviews used in overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks [ 88 ].

Other types of methodological studies

In addition to the previously mentioned experimental methodological studies, there may exist other types of methodological studies not captured here.

What is the design?

Methodological studies that are descriptive

Most methodological studies are purely descriptive and report their findings as counts (percent) and means (standard deviation) or medians (interquartile range). For example, Mbuagbaw et al. described the reporting of research recommendations in Cochrane HIV systematic reviews [ 30 ]. Gohari et al. described the quality of reporting of randomized trials in diabetes in Iran [ 12 ].

Methodological studies that are analytical

Some methodological studies are analytical wherein “analytical studies identify and quantify associations, test hypotheses, identify causes and determine whether an association exists between variables, such as between an exposure and a disease.” [ 89 ] In the case of methodological studies all these investigations are possible. For example, Kosa et al. investigated the association between agreement in primary outcome from trial registry to published manuscript and study covariates. They found that larger and more recent studies were more likely to have agreement [ 15 ]. Tricco et al. compared the conclusion statements from Cochrane and non-Cochrane systematic reviews with a meta-analysis of the primary outcome and found that non-Cochrane reviews were more likely to report positive findings. These results are a test of the null hypothesis that the proportions of Cochrane and non-Cochrane reviews that report positive results are equal [ 90 ].

What is the sampling strategy?

Methodological studies that include the target population

Methodological reviews with narrow research questions may be able to include the entire target population. For example, in the methodological study of Cochrane HIV systematic reviews, Mbuagbaw et al. included all of the available studies ( n  = 103) [ 30 ].

Methodological studies that include a sample of the target population

Many methodological studies use random samples of the target population [ 33 , 91 , 92 ]. Alternatively, purposeful sampling may be used, limiting the sample to a subset of research-related reports published within a certain time period, or in journals with a certain ranking or on a topic. Systematic sampling can also be used when random sampling may be challenging to implement.

What is the unit of analysis?

Methodological studies with a research report as the unit of analysis

Many methodological studies use a research report (e.g. full manuscript of study, abstract portion of the study) as the unit of analysis, and inferences can be made at the study-level. However, both published and unpublished research-related reports can be studied. These may include articles, conference abstracts, registry entries etc.

Methodological studies with a design, analysis or reporting item as the unit of analysis

Some methodological studies report on items which may occur more than once per article. For example, Paquette et al. report on subgroup analyses in Cochrane reviews of atrial fibrillation in which 17 systematic reviews planned 56 subgroup analyses [ 93 ].

This framework is outlined in Fig.  2 .

A proposed framework for methodological studies

Conclusions

Methodological studies have examined different aspects of reporting such as quality, completeness, consistency and adherence to reporting guidelines. As such, many of the methodological study examples cited in this tutorial are related to reporting. However, as an evolving field, the scope of research questions that can be addressed by methodological studies is expected to increase.

In this paper we have outlined the scope and purpose of methodological studies, along with examples of instances in which various approaches have been used. In the absence of formal guidance on the design, conduct, analysis and reporting of methodological studies, we have provided some advice to help make methodological studies consistent. This advice is grounded in good contemporary scientific practice. Generally, the research question should tie in with the sampling approach and planned analysis. We have also highlighted the variables that may inform findings from methodological studies. Lastly, we have provided suggestions for ways in which authors can categorize their methodological studies to inform their design and analysis.

Availability of data and materials

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Abbreviations

Consolidated Standards of Reporting Trials

Evidence, Participants, Intervention, Comparison, Outcome, Timeframe

Grading of Recommendations, Assessment, Development and Evaluations

Participants, Intervention, Comparison, Outcome, Timeframe

Preferred Reporting Items of Systematic reviews and Meta-Analyses

Studies Within a Review

Studies Within a Trial

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

PubMed   Google Scholar  

Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.

PubMed   PubMed Central   Google Scholar  

Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj. 2017;358:j4008.

Lawson DO, Leenus A, Mbuagbaw L. Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review. Pilot Feasibility Studies. 2020;6(1):13.

Puljak L, Makaric ZL, Buljan I, Pieper D. What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. J Comp Eff Res. 2020.

Abbade LPF, Wang M, Sriganesh K, Jin Y, Mbuagbaw L, Thabane L. The framing of research questions using the PICOT format in randomized controlled trials of venous ulcer disease is suboptimal: a systematic survey. Wound Repair Regen. 2017;25(5):892–900.

Gohari F, Baradaran HR, Tabatabaee M, Anijidani S, Mohammadpour Touserkani F, Atlasi R, Razmgir M. Quality of reporting randomized controlled trials (RCTs) in diabetes in Iran; a systematic review. J Diabetes Metab Disord. 2015;15(1):36.

Wang M, Jin Y, Hu ZJ, Thabane A, Dennis B, Gajic-Veljanoski O, Paul J, Thabane L. The reporting quality of abstracts of stepped wedge randomized trials is suboptimal: a systematic survey of the literature. Contemp Clin Trials Commun. 2017;8:1–10.

Shanthanna H, Kaushal A, Mbuagbaw L, Couban R, Busse J, Thabane L: A cross-sectional study of the reporting quality of pilot or feasibility trials in high-impact anesthesia journals Can J Anaesthesia 2018, 65(11):1180–1195.

Kosa SD, Mbuagbaw L, Borg Debono V, Bhandari M, Dennis BB, Ene G, Leenus A, Shi D, Thabane M, Valvasori S, et al. Agreement in reporting between trial publications and current clinical trial registry in high impact journals: a methodological review. Contemporary Clinical Trials. 2018;65:144–50.

Zhang Y, Florez ID, Colunga Lozano LE, Aloweni FAB, Kennedy SA, Li A, Craigie S, Zhang S, Agarwal A, Lopes LC, et al. A systematic survey on reporting and methods for handling missing participant data for continuous outcomes in randomized controlled trials. J Clin Epidemiol. 2017;88:57–66.

CAS   PubMed   Google Scholar  

Hernández AV, Boersma E, Murray GD, Habbema JD, Steyerberg EW. Subgroup analyses in therapeutic cardiovascular clinical trials: are most of them misleading? Am Heart J. 2006;151(2):257–64.

Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, Zhang S, Fruci V, Dennis B, Bawor M, Thabane L. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–88.

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697–703.

Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P. Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary-of-findings tables with a new format. J Clin Epidemiol. 2016;74:7–18.

The Northern Ireland Hub for Trials Methodology Research: SWAT/SWAR Information [ https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/ ]. Accessed 31 Aug 2020.

Chick S, Sánchez P, Ferrin D, Morrice D. How to conduct a successful simulation study. In: Proceedings of the 2003 winter simulation conference: 2003; 2003. p. 66–70.

Google Scholar  

Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987;106(3):485–8.

Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mount Sinai J Med New York. 1996;63(3–4):216–24.

CAS   Google Scholar  

Areia M, Soares M, Dinis-Ribeiro M. Quality reporting of endoscopic diagnostic studies in gastrointestinal journals: where do we stand on the use of the STARD and CONSORT statements? Endoscopy. 2010;42(2):138–47.

Knol M, Groenwold R, Grobbee D. P-values in baseline tables of randomised controlled trials are inappropriate but still common in high impact journals. Eur J Prev Cardiol. 2012;19(2):231–2.

Chen M, Cui J, Zhang AL, Sze DM, Xue CC, May BH. Adherence to CONSORT items in randomized controlled trials of integrative medicine for colorectal Cancer published in Chinese journals. J Altern Complement Med. 2018;24(2):115–24.

Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors' implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;344:e4178.

The Cochrane Methodology Register Issue 2 2009 [ https://cmr.cochrane.org/help.htm ]. Accessed 31 Aug 2020.

Mbuagbaw L, Kredo T, Welch V, Mursleen S, Ross S, Zani B, Motaze NV, Quinlan L. Critical EPICOT items were absent in Cochrane human immunodeficiency virus systematic reviews: a bibliometric analysis. J Clin Epidemiol. 2016;74:66–72.

Barton S, Peckitt C, Sclafani F, Cunningham D, Chau I. The influence of industry sponsorship on the reporting of subgroup analyses within phase III randomised controlled trials in gastrointestinal oncology. Eur J Cancer. 2015;51(18):2732–9.

Setia MS. Methodology series module 5: sampling strategies. Indian J Dermatol. 2016;61(5):505–9.

Wilson B, Burnett P, Moher D, Altman DG, Al-Shahi Salman R. Completeness of reporting of randomised controlled trials including people with transient ischaemic attack or stroke: a systematic review. Eur Stroke J. 2018;3(4):337–46.

Kahale LA, Diab B, Brignardello-Petersen R, Agarwal A, Mustafa RA, Kwong J, Neumann I, Li L, Lopes LC, Briel M, et al. Systematic reviews do not adequately report or address missing outcome data in their analyses: a methodological survey. J Clin Epidemiol. 2018;99:14–23.

De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJPM, et al. Is this clinical trial fully registered?: a statement from the International Committee of Medical Journal Editors*. Ann Intern Med. 2005;143(2):146–8.

Ohtake PJ, Childs JD. Why publish study protocols? Phys Ther. 2014;94(9):1208–9.

Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. A descriptive analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017. BMC Med Res Methodol. 2019;19(1):57.

Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet. 2002;359(9302):248–52.

Porta M (ed.): A dictionary of epidemiology, 5th edn. Oxford: Oxford University Press, Inc.; 2008.

El Dib R, Tikkinen KAO, Akl EA, Gomaa HA, Mustafa RA, Agarwal A, Carpenter CR, Zhang Y, Jorge EC, Almeida R, et al. Systematic survey of randomized trials evaluating the impact of alternative diagnostic strategies on patient-important outcomes. J Clin Epidemiol. 2017;84:61–9.

Helzer JE, Robins LN, Taibleson M, Woodruff RA Jr, Reich T, Wish ED. Reliability of psychiatric diagnosis. I. a methodological review. Arch Gen Psychiatry. 1977;34(2):129–33.

Chung ST, Chacko SK, Sunehag AL, Haymond MW. Measurements of gluconeogenesis and Glycogenolysis: a methodological review. Diabetes. 2015;64(12):3996–4010.

CAS   PubMed   PubMed Central   Google Scholar  

Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med. 2002;21(11):1513–24.

Moen EL, Fricano-Kugler CJ, Luikart BW, O’Malley AJ. Analyzing clustered data: why and how to account for multiple observations nested within a study participant? PLoS One. 2016;11(1):e0146721.

Zyzanski SJ, Flocke SA, Dickinson LM. On the nature and analysis of clustered data. Ann Fam Med. 2004;2(3):199–200.

Mathes T, Klassen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol. 2017;17(1):152.

Bui DDA, Del Fiol G, Hurdle JF, Jonnalagadda S. Extractive text summarization system to aid data extraction from full text in systematic review development. J Biomed Inform. 2016;64:265–72.

Bui DD, Del Fiol G, Jonnalagadda S. PDF text classification to leverage information extraction from publication reports. J Biomed Inform. 2016;61:141–8.

Maticic K, Krnic Martinic M, Puljak L. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Med Res Methodol. 2019;19(1):32.

Speich B. Blinding in surgical randomized clinical trials in 2015. Ann Surg. 2017;266(1):21–2.

Abraha I, Cozzolino F, Orso M, Marchesi M, Germani A, Lombardo G, Eusebi P, De Florio R, Luchetta ML, Iorio A, et al. A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews. J Clin Epidemiol. 2017;84:37–46.

Zhong Y, Zhou W, Jiang H, Fan T, Diao X, Yang H, Min J, Wang G, Fu J, Mao B. Quality of reporting of two-group parallel randomized controlled clinical trials of multi-herb formulae: A survey of reports indexed in the Science Citation Index Expanded. Eur J Integrative Med. 2011;3(4):e309–16.

Farrokhyar F, Chu R, Whitlock R, Thabane L. A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007;50(4):266–77.

Oltean H, Gagnier JJ. Use of clustering analysis in randomized controlled trials in orthopaedic surgery. BMC Med Res Methodol. 2015;15:17.

Fleming PS, Koletsi D, Pandis N. Blinded by PRISMA: are systematic reviewers focusing on PRISMA and ignoring other guidelines? PLoS One. 2014;9(5):e96407.

Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW. Standards of reporting of randomized controlled trials in general surgery: can we do better? Ann Surg. 2006;244(5):663–7.

de Vries TW, van Roon EN. Low quality of reporting adverse drug reactions in paediatric randomised controlled trials. Arch Dis Child. 2010;95(12):1023–6.

Borg Debono V, Zhang S, Ye C, Paul J, Arya A, Hurlburt L, Murthy Y, Thabane L. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiol. 2012;12:13.

Kaiser KA, Cofield SS, Fontaine KR, Glasser SP, Thabane L, Chu R, Ambrale S, Dwary AD, Kumar A, Nayyar G, et al. Is funding source related to study reporting quality in obesity or nutrition randomized control trials in top-tier medical journals? Int J Obes. 2012;36(7):977–81.

Thomas O, Thabane L, Douketis J, Chu R, Westfall AO, Allison DB. Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes. 2008;32(10):1531–6.

Khan NR, Saad H, Oravec CS, Rossi N, Nguyen V, Venable GT, Lillard JC, Patel P, Taylor DR, Vaughn BN, et al. A review of industry funding in randomized controlled trials published in the neurosurgical literature-the elephant in the room. Neurosurgery. 2018;83(5):890–7.

Hansen C, Lundh A, Rasmussen K, Hrobjartsson A. Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst Rev. 2019;8:Mr000047.

Kiehna EN, Starke RM, Pouratian N, Dumont AS. Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg. 2011;114(2):280–5.

Liu LQ, Morris PJ, Pengel LH. Compliance to the CONSORT statement of randomized controlled trials in solid organ transplantation: a 3-year overview. Transpl Int. 2013;26(3):300–6.

Bala MM, Akl EA, Sun X, Bassler D, Mertz D, Mejza F, Vandvik PO, Malaga G, Johnston BC, Dahm P, et al. Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis. J Clin Epidemiol. 2013;66(3):286–95.

Lee SY, Teoh PJ, Camm CF, Agha RA. Compliance of randomized controlled trials in trauma surgery with the CONSORT statement. J Trauma Acute Care Surg. 2013;75(4):562–72.

Ziogas DC, Zintzaras E. Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement. Ann Epidemiol. 2009;19(7):494–500.

Alvarez F, Meyer N, Gourraud PA, Paul C. CONSORT adoption and quality of reporting of randomized controlled trials: a systematic analysis in two dermatology journals. Br J Dermatol. 2009;161(5):1159–65.

Mbuagbaw L, Thabane M, Vanniyasingam T, Borg Debono V, Kosa S, Zhang S, Ye C, Parpia S, Dennis BB, Thabane L. Improvement in the quality of abstracts in major clinical journals since CONSORT extension for abstracts: a systematic review. Contemporary Clin trials. 2014;38(2):245–50.

Thabane L, Chu R, Cuddy K, Douketis J. What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials. Int J Obes. 2007;31(10):1554–9.

Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence Based Med. 2017;22(4):139.

METRIC - MEthodological sTudy ReportIng Checklist: guidelines for reporting methodological studies in health research [ http://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-other-study-designs/#METRIC ]. Accessed 31 Aug 2020.

Jager KJ, Zoccali C, MacLeod A, Dekker FW. Confounding: what it is and how to deal with it. Kidney Int. 2008;73(3):256–60.

Parker SG, Halligan S, Erotocritou M, Wood CPJ, Boulton RW, Plumb AAO, Windsor ACJ, Mallett S. A systematic methodological review of non-randomised interventional studies of elective ventral hernia repair: clear definitions and a standardised minimum dataset are needed. Hernia. 2019.

Bouwmeester W, Zuithoff NPA, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KGM. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1–12.

Schiller P, Burchardi N, Niestroj M, Kieser M. Quality of reporting of clinical non-inferiority and equivalence randomised trials--update and extension. Trials. 2012;13:214.

Riado Minguez D, Kowalski M, Vallve Odena M, Longin Pontzen D, Jelicic Kadic A, Jeric M, Dosenovic S, Jakus D, Vrdoljak M, Poklepovic Pericic T, et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analg. 2017;125(4):1348–54.

Thabut G, Estellat C, Boutron I, Samama CM, Ravaud P. Methodological issues in trials assessing primary prophylaxis of venous thrombo-embolism. Eur Heart J. 2005;27(2):227–36.

Puljak L, Riva N, Parmelli E, González-Lorenzo M, Moja L, Pieper D. Data extraction methods: an analysis of internal reporting discrepancies in single manuscripts and practical advice. J Clin Epidemiol. 2020;117:158–64.

Ritchie A, Seubert L, Clifford R, Perry D, Bond C. Do randomised controlled trials relevant to pharmacy meet best practice standards for quality conduct and reporting? A systematic review. Int J Pharm Pract. 2019.

Babic A, Vuka I, Saric F, Proloscic I, Slapnicar E, Cavar J, Pericic TP, Pieper D, Puljak L. Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent. J Clin Epidemiol. 2019.

Tan A, Porcher R, Crequit P, Ravaud P, Dechartres A. Differences in treatment effect size between overall survival and progression-free survival in immunotherapy trials: a Meta-epidemiologic study of trials with results posted at ClinicalTrials.gov. J Clin Oncol. 2017;35(15):1686–94.

Croitoru D, Huang Y, Kurdina A, Chan AW, Drucker AM. Quality of reporting in systematic reviews published in dermatology journals. Br J Dermatol. 2020;182(6):1469–76.

Khan MS, Ochani RK, Shaikh A, Vaduganathan M, Khan SU, Fatima K, Yamani N, Mandrola J, Doukky R, Krasuski RA: Assessing the Quality of Reporting of Harms in Randomized Controlled Trials Published in High Impact Cardiovascular Journals. Eur Heart J Qual Care Clin Outcomes 2019.

Rosmarakis ES, Soteriades ES, Vergidis PI, Kasiakou SK, Falagas ME. From conference abstract to full paper: differences between data presented in conferences and journals. FASEB J. 2005;19(7):673–80.

Mueller M, D’Addario M, Egger M, Cevallos M, Dekkers O, Mugglin C, Scott P. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol. 2018;18(1):44.

Li G, Abbade LPF, Nwosu I, Jin Y, Leenus A, Maaz M, Wang M, Bhatt M, Zielinski L, Sanger N, et al. A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Med Res Methodol. 2017;17(1):181.

Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203.

Analytical study [ https://medical-dictionary.thefreedictionary.com/analytical+study ]. Accessed 31 Aug 2020.

Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. J Clin Epidemiol. 2009;62(4):380–6 e381.

Schalken N, Rietbergen C. The reporting quality of systematic reviews and Meta-analyses in industrial and organizational psychology: a systematic review. Front Psychol. 2017;8:1395.

Ranker LR, Petersen JM, Fox MP. Awareness of and potential for dependent error in the observational epidemiologic literature: A review. Ann Epidemiol. 2019;36:15–9 e12.

Paquette M, Alotaibi AM, Nieuwlaat R, Santesso N, Mbuagbaw L. A meta-epidemiological study of subgroup analyses in cochrane systematic reviews of atrial fibrillation. Syst Rev. 2019;8(1):241.

Download references

Acknowledgements

This work did not receive any dedicated funding.

Author information

Authors and affiliations.

Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada

Lawrence Mbuagbaw, Daeria O. Lawson & Lehana Thabane

Biostatistics Unit/FSORC, 50 Charlton Avenue East, St Joseph’s Healthcare—Hamilton, 3rd Floor Martha Wing, Room H321, Hamilton, Ontario, L8N 4A6, Canada

Lawrence Mbuagbaw & Lehana Thabane

Centre for the Development of Best Practices in Health, Yaoundé, Cameroon

Lawrence Mbuagbaw

Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia

Livia Puljak

Department of Epidemiology and Biostatistics, School of Public Health – Bloomington, Indiana University, Bloomington, IN, 47405, USA

David B. Allison

Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, ON, Canada

Lehana Thabane

Centre for Evaluation of Medicine, St. Joseph’s Healthcare-Hamilton, Hamilton, ON, Canada

Population Health Research Institute, Hamilton Health Sciences, Hamilton, ON, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

LM conceived the idea and drafted the outline and paper. DOL and LT commented on the idea and draft outline. LM, LP and DOL performed literature searches and data extraction. All authors (LM, DOL, LT, LP, DBA) reviewed several draft versions of the manuscript and approved the final manuscript.

Corresponding author

Correspondence to Lawrence Mbuagbaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

DOL, DBA, LM, LP and LT are involved in the development of a reporting guideline for methodological studies.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mbuagbaw, L., Lawson, D.O., Puljak, L. et al. A tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol 20 , 226 (2020). https://doi.org/10.1186/s12874-020-01107-7

Download citation

Received : 27 May 2020

Accepted : 27 August 2020

Published : 07 September 2020

DOI : https://doi.org/10.1186/s12874-020-01107-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Methodological study
• Meta-epidemiology
• Research methods
• Research-on-research

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Dissertations & projects: Literature-based projects

• Research questions
• The process of reviewing
• Project management
• Literature-based projects

On these pages:

“As a general rule, the introduction is usually around 5 to 10 per cent of the word limit; each chapter around 15 to 25 per cent; and the conclusion around 5 per cent.” Bryan Greetham, How to Write Your Undergraduate Dissertation

This page gives guidance on the structure of a literature-based project.   That is, a project where the data is found in existing literature rather than found through primary research. They may also include information from primary sources such as original documents or other sources.

How to structure a literature-based project

The structure of a literature-based dissertation is usually thematic, but make sure to check with your supervisor to make sure you are abiding by your department’s project specifications. A typical literature-based dissertation will be broken up into the following sections:

Abstract or summary

Acknowledgments, contents page, introduction, themed chapters.

• Bibliography/Reference list

Use this basic structure as your document plan . Remember that you do not need to write it in the order it will finally be written in. 

For more advice on managing the order of your project, see our section on Project Management.   

If you use the template provided on our Formatting page, you will see that it already has a title page included. You just need to fill in the appropriate boxes by typing or choosing from the drop-down-lists. The information you need to provide is: 

• Type of assignment (thesis, dissertation or independent project)
• Partial or full fulfilment information
• Subject area
• Your name (and previous qualifications if applicable)
• Month and year of submission

This may not always be required - check with your tutor.

• It is  independent  of the rest of the report - it is a mini-report, which needs to make sense completely on its own.
• References should  not  be included.
• Nothing should appear in the abstract that is not in the rest of the report.
• Usually between 200-300 words.
• Write as a  single  paragraph.

It is recommended that you write your abstract  after  your report.

If you choose not to use the template, then you will need to go through the document after it is written and create a list showing which heading is on which page of your document.

Purpose: To thank those who were directly involved in your work .

• Do not confuse the acknowledgements section with a dedication - this is not where you thank your friends and relatives unless they have helped you with your manuscript.
• Acknowledgments are about courtesy, where you thank those who were directly involved in your work, or were involved in supporting your work (technicians, tutors, other students, financial support etc).
• This section tends to be  very brief , a few lines at the most. Identify those who provided you with the most support, and thank them appropriately.
• At the very least, make sure you acknowledge your supervisor!!

Purpose: To state the research problem and give a brief introduction to the background literature, provide justification for your research questions and explain your methodology and main findings.

• Explain what the problem you will be addressing is, what your research questions are, and why they will help address the issue.
• Explain (and justify) your methodology - where you searched, what your keywords were, what your inclusion and exclusion criteria were,
• Define the scope of the dissertation, explaining any limitations.
• Lay out the structure of the dissertation, taking the reader through each section and providing any key definitions.
• Very briefly describe what your main findings are - but leave the detail for the sections below.

It is good practice to come back to the introduction after you have finished writing up the rest of the document to ensure it sets the appropriately scene for subsequent sections.

Should you have a separate literature review chapter?

Not usually , as your project is basically a big literature review, it isn't necessary to have a separate chapter. You would normally introduce background literature in your introduction instead.

However, if your supervisor suggests a separate chapter then it could go at this point, after the main introduction (which would then not include background literature). 

For more advice on writing a literature review see the Literature Review pages on this guide.

Purpose: To present the themes you have identified in your research and explain how they contribute to answering your research questions

You will typically have 3-5 themed chapters. Each one should contain:

• An introduction to the theme - what things it means and what it incorporates.
• How the theme was addressed within the literature - this should be analytical not just descriptive.
• A conclusion which shows how the theme relates to the research question(s).

Ensuring your themed chapters flow

Choosing the order of your theme chapters is an important part of the structure to your project. For example, if you study History and your project covers a topic that develops over a large time period, it may be best to order each chapter chronologically. Other subjects may have a natural narrative running through the themes. Think about how your reader will be able to follow along with your overall argument.

Although each chapter must be dedicated to a particular theme, it must link back to previous chapters and flow into the following chapter. You need to ensure they do not seem like they are unrelated to each other. There will be overlaps, mention these.

Some literature-based projects will focus on primary sources. If yours does, make sure primary sources are at the core of your paragraphs and chapters, and use secondary sources to expand and explore the theme further. 

Purpose: To present the conclusion that you have reached as a result of both the background literature review and the analysis in your thematic chapters

A conclusion summarises all the points you have previously made and it  should not  include any evidence or topics you have not included in your introduction or themed chapters. There should be no surprises.

It should be about 5-10% of your word limit so make sure you leave enough words to do it justice. There will be marks in the marking scheme specifically allocated to the strength of your conclusion which cannot be made up elsewhere.

Some conclusions will also include recommendations for practice or ideas for further research. Check with your supervisor to see if they are expecting either or both of these.

Reference list

It is good practice to develop a reference list whilst  writing the project, rather than leaving it until the end. This prevents a lot of searching around trying to remember where you accessed a particular source. If using primary sources, it also allows you to monitor the balance between primary and secondary sources included in the project. There is software available to help manage your references and the university officially supports RefWorks and EndNote. 

For more advice on reference management, see our Skills Guide: Referencing Software

• Transcriptions
• Correspondence
• Ethical approval forms

If you have information that you would like to include but are finding it disrupts the main body of text as its too cumbersome, or would distract from the main arguments of your dissertation, the information can be included in the appendix section. Each appendix should be focused on one item. 

Appendices  should not include any information that is key to your topic or overall argument. 

• << Previous: Structure
• Next: Scientific >>
• Last Updated: Apr 24, 2024 1:09 PM
• URL: https://libguides.hull.ac.uk/dissertations
• Login to LibApps
• Library websites Privacy Policy
• University of Hull privacy policy & cookies
• Website terms and conditions
• Accessibility
• Report a problem

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What Is a Research Methodology? | Steps & Tips

What Is a Research Methodology? | Steps & Tips

Published on 25 February 2019 by Shona McCombes . Revised on 10 October 2022.

Your research methodology discusses and explains the data collection and analysis methods you used in your research. A key part of your thesis, dissertation, or research paper, the methodology chapter explains what you did and how you did it, allowing readers to evaluate the reliability and validity of your research.

It should include:

• The type of research you conducted
• How you collected and analysed your data
• Any tools or materials you used in the research
• Why you chose these methods
• Your methodology section should generally be written in the past tense .
• Academic style guides in your field may provide detailed guidelines on what to include for different types of studies.
• Your citation style might provide guidelines for your methodology section (e.g., an APA Style methods section ).

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

How to write a research methodology, why is a methods section important, step 1: explain your methodological approach, step 2: describe your data collection methods, step 3: describe your analysis method, step 4: evaluate and justify the methodological choices you made, tips for writing a strong methodology chapter, frequently asked questions about methodology.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Your methods section is your opportunity to share how you conducted your research and why you chose the methods you chose. It’s also the place to show that your research was rigorously conducted and can be replicated .

It gives your research legitimacy and situates it within your field, and also gives your readers a place to refer to if they have any questions or critiques in other sections.

You can start by introducing your overall approach to your research. You have two options here.

Option 1: Start with your “what”

What research problem or question did you investigate?

• Aim to describe the characteristics of something?
• Explore an under-researched topic?
• Establish a causal relationship?

And what type of data did you need to achieve this aim?

• Quantitative data , qualitative data , or a mix of both?
• Primary data collected yourself, or secondary data collected by someone else?
• Experimental data gathered by controlling and manipulating variables, or descriptive data gathered via observations?

Option 2: Start with your “why”

Depending on your discipline, you can also start with a discussion of the rationale and assumptions underpinning your methodology. In other words, why did you choose these methods for your study?

• Why is this the best way to answer your research question?
• Is this a standard methodology in your field, or does it require justification?
• Were there any ethical considerations involved in your choices?
• What are the criteria for validity and reliability in this type of research ?

Once you have introduced your reader to your methodological approach, you should share full details about your data collection methods .

Quantitative methods

In order to be considered generalisable, you should describe quantitative research methods in enough detail for another researcher to replicate your study.

Here, explain how you operationalised your concepts and measured your variables. Discuss your sampling method or inclusion/exclusion criteria, as well as any tools, procedures, and materials you used to gather your data.

Surveys Describe where, when, and how the survey was conducted.

• How did you design the questionnaire?
• What form did your questions take (e.g., multiple choice, Likert scale )?
• Were your surveys conducted in-person or virtually?
• What sampling method did you use to select participants?
• What was your sample size and response rate?

Experiments Share full details of the tools, techniques, and procedures you used to conduct your experiment.

• How did you design the experiment ?
• How did you recruit participants?
• How did you manipulate and measure the variables ?
• What tools did you use?

Existing data Explain how you gathered and selected the material (such as datasets or archival data) that you used in your analysis.

• Where did you source the material?
• How was the data originally produced?
• What criteria did you use to select material (e.g., date range)?

The survey consisted of 5 multiple-choice questions and 10 questions measured on a 7-point Likert scale.

The goal was to collect survey responses from 350 customers visiting the fitness apparel company’s brick-and-mortar location in Boston on 4–8 July 2022, between 11:00 and 15:00.

Here, a customer was defined as a person who had purchased a product from the company on the day they took the survey. Participants were given 5 minutes to fill in the survey anonymously. In total, 408 customers responded, but not all surveys were fully completed. Due to this, 371 survey results were included in the analysis.

Qualitative methods

In qualitative research , methods are often more flexible and subjective. For this reason, it’s crucial to robustly explain the methodology choices you made.

Be sure to discuss the criteria you used to select your data, the context in which your research was conducted, and the role you played in collecting your data (e.g., were you an active participant, or a passive observer?)

Interviews or focus groups Describe where, when, and how the interviews were conducted.

• How did you find and select participants?
• How many participants took part?
• What form did the interviews take ( structured , semi-structured , or unstructured )?
• How long were the interviews?
• How were they recorded?

Participant observation Describe where, when, and how you conducted the observation or ethnography .

• What group or community did you observe? How long did you spend there?
• How did you gain access to this group? What role did you play in the community?
• How long did you spend conducting the research? Where was it located?
• How did you record your data (e.g., audiovisual recordings, note-taking)?

Existing data Explain how you selected case study materials for your analysis.

• What type of materials did you analyse?
• How did you select them?

In order to gain better insight into possibilities for future improvement of the fitness shop’s product range, semi-structured interviews were conducted with 8 returning customers.

Here, a returning customer was defined as someone who usually bought products at least twice a week from the store.

Surveys were used to select participants. Interviews were conducted in a small office next to the cash register and lasted approximately 20 minutes each. Answers were recorded by note-taking, and seven interviews were also filmed with consent. One interviewee preferred not to be filmed.

Mixed methods

Mixed methods research combines quantitative and qualitative approaches. If a standalone quantitative or qualitative study is insufficient to answer your research question, mixed methods may be a good fit for you.

Mixed methods are less common than standalone analyses, largely because they require a great deal of effort to pull off successfully. If you choose to pursue mixed methods, it’s especially important to robustly justify your methods here.

Prevent plagiarism, run a free check.

Next, you should indicate how you processed and analysed your data. Avoid going into too much detail: you should not start introducing or discussing any of your results at this stage.

In quantitative research , your analysis will be based on numbers. In your methods section, you can include:

• How you prepared the data before analysing it (e.g., checking for missing data , removing outliers , transforming variables)
• Which software you used (e.g., SPSS, Stata or R)
• Which statistical tests you used (e.g., two-tailed t test , simple linear regression )

In qualitative research, your analysis will be based on language, images, and observations (often involving some form of textual analysis ).

Specific methods might include:

• Content analysis : Categorising and discussing the meaning of words, phrases and sentences
• Thematic analysis : Coding and closely examining the data to identify broad themes and patterns
• Discourse analysis : Studying communication and meaning in relation to their social context

Mixed methods combine the above two research methods, integrating both qualitative and quantitative approaches into one coherent analytical process.

Above all, your methodology section should clearly make the case for why you chose the methods you did. This is especially true if you did not take the most standard approach to your topic. In this case, discuss why other methods were not suitable for your objectives, and show how this approach contributes new knowledge or understanding.

In any case, it should be overwhelmingly clear to your reader that you set yourself up for success in terms of your methodology’s design. Show how your methods should lead to results that are valid and reliable, while leaving the analysis of the meaning, importance, and relevance of your results for your discussion section .

• Quantitative: Lab-based experiments cannot always accurately simulate real-life situations and behaviours, but they are effective for testing causal relationships between variables .
• Qualitative: Unstructured interviews usually produce results that cannot be generalised beyond the sample group , but they provide a more in-depth understanding of participants’ perceptions, motivations, and emotions.
• Mixed methods: Despite issues systematically comparing differing types of data, a solely quantitative study would not sufficiently incorporate the lived experience of each participant, while a solely qualitative study would be insufficiently generalisable.

Remember that your aim is not just to describe your methods, but to show how and why you applied them. Again, it’s critical to demonstrate that your research was rigorously conducted and can be replicated.

1. Focus on your objectives and research questions

The methodology section should clearly show why your methods suit your objectives  and convince the reader that you chose the best possible approach to answering your problem statement and research questions .

2. Cite relevant sources

Your methodology can be strengthened by referencing existing research in your field. This can help you to:

• Show that you followed established practice for your type of research
• Discuss how you decided on your approach by evaluating existing research
• Present a novel methodological approach to address a gap in the literature

3. Write for your audience

Consider how much information you need to give, and avoid getting too lengthy. If you are using methods that are standard for your discipline, you probably don’t need to give a lot of background or justification.

Regardless, your methodology should be a clear, well-structured text that makes an argument for your approach, not just a list of technical details and procedures.

Methodology refers to the overarching strategy and rationale of your research. Developing your methodology involves studying the research methods used in your field and the theories or principles that underpin them, in order to choose the approach that best matches your objectives.

Methods are the specific tools and procedures you use to collect and analyse data (e.g. interviews, experiments , surveys , statistical tests ).

In a dissertation or scientific paper, the methodology chapter or methods section comes after the introduction and before the results , discussion and conclusion .

Depending on the length and type of document, you might also include a literature review or theoretical framework before the methodology.

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to test a hypothesis by systematically collecting and analysing data, while qualitative methods allow you to explore ideas and experiences in depth.

A sample is a subset of individuals from a larger population. Sampling means selecting the group that you will actually collect data from in your research.

For example, if you are researching the opinions of students in your university, you could survey a sample of 100 students.

Statistical sampling allows you to test a hypothesis about the characteristics of a population. There are various sampling methods you can use to ensure that your sample is representative of the population as a whole.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, October 10). What Is a Research Methodology? | Steps & Tips. Scribbr. Retrieved 14 May 2024, from https://www.scribbr.co.uk/thesis-dissertation/methodology/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a literature review | guide, template, & examples, what is a theoretical framework | a step-by-step guide.

Library Guides

Dissertations 4: methodology: methods.

• Introduction & Philosophy
• Methodology

Primary & Secondary Sources, Primary & Secondary Data

When describing your research methods, you can start by stating what kind of secondary and, if applicable, primary sources you used in your research. Explain why you chose such sources, how well they served your research, and identify possible issues encountered using these sources.  

Definitions  

There is some confusion on the use of the terms primary and secondary sources, and primary and secondary data. The confusion is also due to disciplinary differences (Lombard 2010). Whilst you are advised to consult the research methods literature in your field, we can generalise as follows:  

Secondary sources 

Secondary sources normally include the literature (books and articles) with the experts' findings, analysis and discussions on a certain topic (Cottrell, 2014, p123). Secondary sources often interpret primary sources.  

Primary sources 

Primary sources are "first-hand" information such as raw data, statistics, interviews, surveys, law statutes and law cases. Even literary texts, pictures and films can be primary sources if they are the object of research (rather than, for example, documentaries reporting on something else, in which case they would be secondary sources). The distinction between primary and secondary sources sometimes lies on the use you make of them (Cottrell, 2014, p123). 

Primary data 

Primary data are data (primary sources) you directly obtained through your empirical work (Saunders, Lewis and Thornhill 2015, p316). 

Secondary data 

Secondary data are data (primary sources) that were originally collected by someone else (Saunders, Lewis and Thornhill 2015, p316).   

Comparison between primary and secondary data   

Use  

Virtually all research will use secondary sources, at least as background information. 

Often, especially at the postgraduate level, it will also use primary sources - secondary and/or primary data. The engagement with primary sources is generally appreciated, as less reliant on others' interpretations, and closer to 'facts'. 

The use of primary data, as opposed to secondary data, demonstrates the researcher's effort to do empirical work and find evidence to answer her specific research question and fulfill her specific research objectives. Thus, primary data contribute to the originality of the research.    

Ultimately, you should state in this section of the methodology: 

What sources and data you are using and why (how are they going to help you answer the research question and/or test the hypothesis. 

If using primary data, why you employed certain strategies to collect them. 

What the advantages and disadvantages of your strategies to collect the data (also refer to the research in you field and research methods literature). 

Quantitative, Qualitative & Mixed Methods

The methodology chapter should reference your use of quantitative research, qualitative research and/or mixed methods. The following is a description of each along with their advantages and disadvantages. 

Quantitative research 

Quantitative research uses numerical data (quantities) deriving, for example, from experiments, closed questions in surveys, questionnaires, structured interviews or published data sets (Cottrell, 2014, p93). It normally processes and analyses this data using quantitative analysis techniques like tables, graphs and statistics to explore, present and examine relationships and trends within the data (Saunders, Lewis and Thornhill, 2015, p496). 

Qualitative research  

Qualitative research is generally undertaken to study human behaviour and psyche. It uses methods like in-depth case studies, open-ended survey questions, unstructured interviews, focus groups, or unstructured observations (Cottrell, 2014, p93). The nature of the data is subjective, and also the analysis of the researcher involves a degree of subjective interpretation. Subjectivity can be controlled for in the research design, or has to be acknowledged as a feature of the research. Subject-specific books on (qualitative) research methods offer guidance on such research designs.  

Mixed methods 

Mixed-method approaches combine both qualitative and quantitative methods, and therefore combine the strengths of both types of research. Mixed methods have gained popularity in recent years.  

When undertaking mixed-methods research you can collect the qualitative and quantitative data either concurrently or sequentially. If sequentially, you can for example, start with a few semi-structured interviews, providing qualitative insights, and then design a questionnaire to obtain quantitative evidence that your qualitative findings can also apply to a wider population (Specht, 2019, p138). 

Ultimately, your methodology chapter should state: 

Whether you used quantitative research, qualitative research or mixed methods. 

Why you chose such methods (and refer to research method sources). 

Why you rejected other methods. 

How well the method served your research. 

The problems or limitations you encountered. 

Doug Specht, Senior Lecturer at the Westminster School of Media and Communication, explains mixed methods research in the following video:

LinkedIn Learning Video on Academic Research Foundations: Quantitative

The video covers the characteristics of quantitative research, and explains how to approach different parts of the research process, such as creating a solid research question and developing a literature review. He goes over the elements of a study, explains how to collect and analyze data, and shows how to present your data in written and numeric form.

Link to quantitative research video

Some Types of Methods

There are several methods you can use to get primary data. To reiterate, the choice of the methods should depend on your research question/hypothesis. 

Whatever methods you will use, you will need to consider: 

why did you choose one technique over another? What were the advantages and disadvantages of the technique you chose? 

what was the size of your sample? Who made up your sample? How did you select your sample population? Why did you choose that particular sampling strategy?) 

ethical considerations (see also tab...)  

safety considerations  

validity  

feasibility  

recording  

procedure of the research (see box procedural method...).  

Check Stella Cottrell's book  Dissertations and Project Reports: A Step by Step Guide  for some succinct yet comprehensive information on most methods (the following account draws mostly on her work). Check a research methods book in your discipline for more specific guidance.  

Experiments 

Experiments are useful to investigate cause and effect, when the variables can be tightly controlled. They can test a theory or hypothesis in controlled conditions. Experiments do not prove or disprove an hypothesis, instead they support or not support an hypothesis. When using the empirical and inductive method it is not possible to achieve conclusive results. The results may only be valid until falsified by other experiments and observations. 

For more information on Scientific Method, click here . 

Observations 

Observational methods are useful for in-depth analyses of behaviours in people, animals, organisations, events or phenomena. They can test a theory or products in real life or simulated settings. They generally a qualitative research method.  

Questionnaires and surveys 

Questionnaires and surveys are useful to gain opinions, attitudes, preferences, understandings on certain matters. They can provide quantitative data that can be collated systematically; qualitative data, if they include opportunities for open-ended responses; or both qualitative and quantitative elements. 

Interviews  

Interviews are useful to gain rich, qualitative information about individuals' experiences, attitudes or perspectives. With interviews you can follow up immediately on responses for clarification or further details. There are three main types of interviews: structured (following a strict pattern of questions, which expect short answers), semi-structured (following a list of questions, with the opportunity to follow up the answers with improvised questions), and unstructured (following a short list of broad questions, where the respondent can lead more the conversation) (Specht, 2019, p142). 

This short video on qualitative interviews discusses best practices and covers qualitative interview design, preparation and data collection methods. 

Focus groups   

In this case, a group of people (normally, 4-12) is gathered for an interview where the interviewer asks questions to such group of participants. Group interactions and discussions can be highly productive, but the researcher has to beware of the group effect, whereby certain participants and views dominate the interview (Saunders, Lewis and Thornhill 2015, p419). The researcher can try to minimise this by encouraging involvement of all participants and promoting a multiplicity of views. 

This video focuses on strategies for conducting research using focus groups.  

Check out the guidance on online focus groups by Aliaksandr Herasimenka, which is attached at the bottom of this text box. 

Case study 

Case studies are often a convenient way to narrow the focus of your research by studying how a theory or literature fares with regard to a specific person, group, organisation, event or other type of entity or phenomenon you identify. Case studies can be researched using other methods, including those described in this section. Case studies give in-depth insights on the particular reality that has been examined, but may not be representative of what happens in general, they may not be generalisable, and may not be relevant to other contexts. These limitations have to be acknowledged by the researcher.     

Content analysis 

Content analysis consists in the study of words or images within a text. In its broad definition, texts include books, articles, essays, historical documents, speeches, conversations, advertising, interviews, social media posts, films, theatre, paintings or other visuals. Content analysis can be quantitative (e.g. word frequency) or qualitative (e.g. analysing intention and implications of the communication). It can detect propaganda, identify intentions of writers, and can see differences in types of communication (Specht, 2019, p146). Check this page on collecting, cleaning and visualising Twitter data.

Extra links and resources:  

Research Methods  

A clear and comprehensive overview of research methods by Emerald Publishing. It includes: crowdsourcing as a research tool; mixed methods research; case study; discourse analysis; ground theory; repertory grid; ethnographic method and participant observation; interviews; focus group; action research; analysis of qualitative data; survey design; questionnaires; statistics; experiments; empirical research; literature review; secondary data and archival materials; data collection. 

Doing your dissertation during the COVID-19 pandemic  

Resources providing guidance on doing dissertation research during the pandemic: Online research methods; Secondary data sources; Webinars, conferences and podcasts; 

• Virtual Focus Groups Guidance on managing virtual focus groups

5 Minute Methods Videos

The following are a series of useful videos that introduce research methods in five minutes. These resources have been produced by lecturers and students with the University of Westminster's School of Media and Communication. 

Case Study Research

Research Ethics

Quantitative Content Analysis 

Sequential Analysis 

Qualitative Content Analysis 

Thematic Analysis 

Social Media Research 

Mixed Method Research 

Procedural Method

In this part, provide an accurate, detailed account of the methods and procedures that were used in the study or the experiment (if applicable!). 

Include specifics about participants, sample, materials, design and methods. 

If the research involves human subjects, then include a detailed description of who and how many participated along with how the participants were selected.  

Describe all materials used for the study, including equipment, written materials and testing instruments. 

Identify the study's design and any variables or controls employed. 

Write out the steps in the order that they were completed. 

Indicate what participants were asked to do, how measurements were taken and any calculations made to raw data collected. 

Specify statistical techniques applied to the data to reach your conclusions. 

Provide evidence that you incorporated rigor into your research. This is the quality of being thorough and accurate and considers the logic behind your research design. 

Highlight any drawbacks that may have limited your ability to conduct your research thoroughly. 

You have to provide details to allow others to replicate the experiment and/or verify the data, to test the validity of the research. 

Bibliography

Cottrell, S. (2014). Dissertations and project reports: a step by step guide. Hampshire, England: Palgrave Macmillan.

Lombard, E. (2010). Primary and secondary sources.  The Journal of Academic Librarianship , 36(3), 250-253

Saunders, M.N.K., Lewis, P. and Thornhill, A. (2015).  Research Methods for Business Students.  New York: Pearson Education. 

Specht, D. (2019).  The Media And Communications Study Skills Student Guide . London: University of Westminster Press.  

• << Previous: Introduction & Philosophy
• Next: Ethics >>
• Last Updated: Sep 14, 2022 12:58 PM
• URL: https://libguides.westminster.ac.uk/methodology-for-dissertations

CONNECT WITH US

• Open access
• Published: 10 May 2024

Community-based participatory-research through co-design: supporting collaboration from all sides of disability

• Cloe Benz   ORCID: orcid.org/0000-0001-6950-8855 1 ,
• Will Scott-Jeffs 2 ,
• K. A. McKercher   ORCID: orcid.org/0000-0003-4417-585X 3 ,
• Mai Welsh   ORCID: orcid.org/0000-0002-7818-0115 2 , 4 ,
• Richard Norman   ORCID: orcid.org/0000-0002-3112-3893 1 ,
• Delia Hendrie   ORCID: orcid.org/0000-0001-5022-5281 1 ,
• Matthew Locantro 2 &
• Suzanne Robinson   ORCID: orcid.org/0000-0001-5703-6475 1 , 5  

Research Involvement and Engagement volume  10 , Article number:  47 ( 2024 ) Cite this article

607 Accesses

Metrics details

As co-design and community-based participatory research gain traction in health and disability, the challenges and benefits of collaboratively conducting research need to be considered. Current literature supports using co-design to improve service quality and create more satisfactory services. However, while the ‘why’ of using co-design is well understood, there is limited literature on ‘ how ’ to co-design. We aimed to describe the application of co-design from start to finish within a specific case study and to reflect on the challenges and benefits created by specific process design choices.

A telepractice re-design project has been a case study example of co-design. The co-design was co-facilitated by an embedded researcher and a peer researcher with lived experience of disability. Embedded in a Western Australian disability organisation, the co-design process included five workshops and a reflection session with a team of 10 lived experience and staff participants (referred to as co-designers) to produce a prototype telepractice model for testing.

The findings are divided into two components. The first describes the process design choices made throughout the co-design implementation case study. This is followed by a reflection on the benefits and challenges resulting from specific process design choices. The reflective process describes the co-designers’ perspective and the researcher’s and organisational experiences. Reflections of the co-designers include balancing idealism and realism, the value of small groups, ensuring accessibility and choice, and learning new skills and gaining new insights. The organisational and research-focused reflections included challenges between time for building relationships and the schedules of academic and organisational decision-making, the messiness of co-design juxtaposed with the processes of ethics applications, and the need for inclusive dissemination of findings.

Conclusions

The authors advocate that co-design is a useful and outcome-generating methodology that proactively enables the inclusion of people with disability and service providers through community-based participatory research and action. Through our experiences, we recommend community-based participatory research, specifically co-design, to generate creative thinking and service design.

Plain language summary

Making better services with communities (called co-design) and doing research with communities (e.g. community-based participatory research) are ways to include people with lived experience in developing and improving the services they use. Academic evidence shows why co-design is valuable, and co-design is increasing in popularity. However, there needs to be more information on how to do co-design. This article describes the process of doing co-design to make telepractice better with a group of lived experience experts and staff at a disability organisation. The co-design process was co-facilitated by two researchers – one with a health background and one with lived experience of disability. Telepractice provides clinical services (such as physiotherapy or nursing) using video calls and other digital technology. The co-design team did five workshops and then reflected on the success of those workshops. Based on the groups’ feedback, the article describes what worked and what was hard according to the co-designers and from the perspective of the researchers and the disability organisation. Topics discussed include the challenge of balancing ideas with realistic expectations, the value of small groups, accessibility and choice opportunities and learning new skills and insights. The research and organisational topics include the need to take time and how that doesn’t fit neatly with academic and business schedules, how the messiness of co-design can clash with approval processes, and different ways of telling people about the project that are more inclusive than traditional research. The authors conclude that co-design and community-based participatory research go well together in including people with lived experience in re-designing services they use.

Peer Review reports

Introduction

Co-design has the potential to positively impact co-designers and their community, researchers, and organisations. Co-design is defined as designing with, not for, people [ 1 ] and can reinvigorate business-as-usual processes, leading to new ideas in industry, community and academia. As co-design and community-based participatory research gain traction, the challenges and benefits of collaborative research between people with lived experience and organisations must be considered [ 2 ].

Disability and healthcare providers previously made decisions for individuals as passive targets of an intervention [ 3 ]. By contrast, the involvement of consumers in their care [ 4 ] has been included as part of accreditation processes [ 4 ] and shown to improve outcomes and satisfaction. For research to sufficiently translate into practice, consumers and providers should be involved actively, not passively [ 4 , 5 ].

Approaches such as community-based participatory research promote “a collaborative approach that equitably involves community members, organisational representatives and researchers in all aspects of the research process” [ 6 ] (page 1). This approach originated in public health research and claims to empower all participants to have a stake in project success, facilitating a more active integration of research into practice and decreasing the knowledge to practice gap 6 . Patient and public involvement (PPI) increases the probability that research focus, community priorities and clinical problems align, which is increasingly demanded by research funders and health systems [ 7 ].

As community-based participatory research is an overarching approach to conducting research, it requires a complementary method, such as co-production, to achieve its aims. Co-production has been attributed to the work of Ostrom et al. [ 8 ], with the term co-design falling under the co-production umbrella. However, co-design can be traced back to the participatory design movement [ 9 ]. The term co-production in the context of this article includes co-planning, co-discovery, co-design, co-delivery, and co-evaluation [ 10 ]. Within this framework, the concept of co-design delineates the collaborative process of discovery, creating, ideating and prototyping to design or redesign an output [ 11 ]. The four principles of co-design, as per McKercher [ 1 ], are sharing power, prioritising relationships, using participatory means and building capacity [ 1 ]. This specific method of co-design [ 1 ] has been used across multiple social and healthcare publications [ 10 , 12 , 13 , 14 ].

A systematic review by Ramos et al. [ 15 ] describes the benefits of co-design in a community-based participatory-research approach, including improved quality and more satisfactory services. However, as identified by Rahman et al. [ 16 ], the ‘ why ’ is well known, but there is limited knowledge of ‘ how ’ to co-design. Multiple articles provide high-level descriptions of workshops or briefly mention the co-design process [ 13 , 17 , 18 , 19 ]. Pearce et al. [ 5 ] include an in-depth table of activities across an entire co-creation process, however within each part i.e., co-design, limited descriptions were included. A recent publication by Marwaa et al. [ 20 ] provides an in-depth description of two workshops focused on product development, and Tariq et al. [ 21 ] provides details of the process of co-designing a research agenda. Davis et al. [ 11 ] discuss co-design workshop delivery strategies summarised across multiple studies without articulating the process from start to finish. Finally, Abimbola et al. [ 22 ] provided the most comprehensive description of a co-design process, including a timeline of events and activities; however, this project only involved clinical staff and did not include community-based participation.

As “We know the why, but we need to know the how-to” [ 16 ] (page 2), of co-design, our primary aim was to describe the application of co-design from start to finish within a specific case study. Our secondary aim was to reflect on the challenges and benefits created by specific process design choices and to provide recommendations for future applications of co-design.

Overview of telepractice project

The case study, a telepractice redesign project, was based at Rocky Bay, a disability support service provider in Perth, Australia [ 23 ]. The project aimed to understand the strengths and pain points of telepractice within Rocky Bay. We expanded this to include telepractice in the wider Australian disability sector. The project also aimed to establish potential improvements to increase the uptake and sustainability of Rocky Bay’s telepractice service into the future. Rocky Bay predominantly serves people under the Australian National Disability Insurance Scheme (NDIS) [ 24 ] by providing a variety of services, including allied health (e.g. physiotherapy, dietetics, speech pathology, etc.), nursing care (including continence and wound care), behaviour support and support coordination [ 23 ]—Rocky Bay services metropolitan Perth and regional Western Australia [ 23 ].

The first author, CB, predominantly conducted this research through an embedded researcher model [ 25 ] between Curtin University and Rocky Bay. An embedded researcher has been defined as “those who work inside host organisations as members of staff while also maintaining an affiliation with an academic institution” [ 25 ] (page 1). They had some prior contextual understanding which stemmed from being a physiotherapist who had previously delivered telehealth in an acute health setting. A peer researcher, WSJ, with lived experience of disability, worked alongside CB. They had no previous experience in research or co-design, this was their first paid employment and they had an interest in digital technology. Peer Researcher is a broad term describing the inclusion of a priority group or social network member as part of the research team to enhance the depth of understanding of the communities to which they belong [ 26 ]. Including a peer researcher in the team promoted equity, collective ownership, and better framing of the research findings to assist with connecting with people with lived experience. These outcomes align with key components of community-based participatory research and co-design [ 27 , 28 , 29 , 30 ].

Person-first language was used as the preference of experts with lived experience who contributed to this research to respect and affirm their identity. However, we respect the right to choose and the potential for others to prefer identity-first language [ 31 ].

A summary of the structure of the phases completed before co-design workshops are represented in Fig.  1 below. Ethical approval for the project was received iteratively before each phase on the timeline (Fig.  1 ) from the Curtin Human Research Ethics Committee (HRE2021-0731). The reporting of this article has been completed in line with the Guidance for Reporting Involvement of Patients and the Public (GRIPP2) checklist [ 7 ].

Summary of telepractice co-design project structure [ 1 ]

Here, we present an outline of the chosen research methods with descriptions of each process design choice and supporting reasons and examples specific to the study. The format is in chronological order, with further details of each step provided in Appendix 1 (Supplementary Material 1).

Methods and results

Process of co-production and preparation for co-design.

Co-production was chosen as the planning method for the study, as the inclusion of community members (Rocky Bay Lived experience experts and Staff) in each step of the research process would increase buy-in and make the research more likely to meet their needs [ 5 ]. An example of co-planning (part of co-production) includes the study steering committee, with a lived experience expert, clinician and project sponsor representatives collaborating on the selection of study aim, methods and recruitment processes. Another example of co-planning, co-design, and co-delivery was recruiting a peer researcher with disability, who worked with the embedded researcher throughout the study design and delivery.

The second process design choice was to attempt to build safe enough conditions for community participation, as people who feel unsafe or unwelcome are less likely to be able to participate fully in the research [ 1 ]. Building conditions for safety was applied by repeatedly acknowledging power imbalances, holding space for community input, and anticipating and offering accessibility adjustments without judgment.

Getting started

Understanding and synthesising what is already known about telepractice experiences and learning from lived experience was prioritised as the first step in the process. We paired a scoping review of the literature with scoping the lived experiences of the community [ 32 ]. Our reasoning was to understand whether the findings aligned and, secondly, to learn what had already been done and to ask what was next, rather than starting from the beginning [ 1 ]. Examples of strategies used in this step included interviewing clinicians and service provider Managers across Australia to establish how they implemented telepractice during the pandemic and understand their views of what worked and what did not. The second learning process occurred onsite at Rocky Bay, with people with lived experience, clinicians and other support staff, whom the embedded researcher and peer researcher interviewed to understand experiences of telepractice at Rocky Bay.

The authors presented the interview findings during focus groups with Rocky Bay participants to share the learnings and confirm we had understood them correctly. The groups were divided into staff and lived experience cohorts, allowing for peer discussions and sharing of common experiences. This helped build relationships and a sense of familiarity moving into the workshop series.

Co-design workshops

This section outlines specific components of the co-design workshop preparation before describing each of the five workshops and the final reflection session.

Staff and community co-designers

Two process design choices were implemented to form the co-design group. The first was to prioritise lived experience input as there are generally fewer opportunities for lived experience leadership in service design [ 16 ], and because the disability community have demanded they be included where the focus impacts them [ 33 ]. To acknowledge the asymmetry of power between community members, people with lived experience of disability and professionals, we ensured the co-design group had at least the same number of lived experience experts as staff.

The second priority for the co-design group was to include people for whom involvement can be difficult to access (e.g. people who are isolated for health reasons and cannot attend in-person sessions, people who live in supported accommodation, part-time staff, and people navigating the dual-role of staff member while disclosing lived experience). It was important to learn from perspectives not commonly heard from and support equity of access for participants [ 4 ].

Workshop series structure

When structuring the workshop series, lived experience co-designers nominated meeting times outside standard work hours to reduce the impact of co-design on work commitments and loss of income while participating. The workshops were designed to be delivered as a hybrid of in-person and online to give co-designers a choice on how they wanted to interact. The workshops were designed as a series of five sequential 90-minute workshops, where co-designers voted for the first workshop to be predominantly in-person and the remainder of the workshops online. Some co-designers chose to attend the initial session in person to build rapport. However, the virtual option remained available. The subsequent online sessions reduced the travel burden on co-designers, which the co-designers prioritised over further face-to-face meetings.

Workshop facilitators

To maintain familiarity and ensure predictability for co-designers, the workshops were co-facilitated by the embedded researcher and peer researcher. The co-facilitators built on relationships formed through previous interactions (interviews and focus groups), and each facilitator represented part of the co-designer group as a clinician or a person with disability. An extra support person was tasked with supporting the co-designers with disability to break down tasks and increase the accessibility of activities. The reason for selecting the support person was that they could contribute their skills as a school teacher to support the communication and completion of activities, and they had no previous experience with disability services to influence the co-designers opinions. This role was adapted from the provocateur role described by McKercher [ 1 ].

Pre-workshop preparations

To prepare for the workshops, each co-designer was asked to complete a brief survey to ensure the co-facilitators understood co-designers collect preferences and needs ahead of the session to enable preparation and make accommodations. The survey included pronouns, accessibility needs and refreshment preferences. Following the survey, the co-facilitators distributed a welcome video; the peer researcher, a familiar person, was videoed explaining what to expect, what not to expect and expected behaviours for the group to support a safe environment [ 1 ]. This process design choice was made to allow co-designers to alleviate any potential anxieties due to not having enough information and to increase predictability.

Workshop resources and supports

As the first workshop was in-person, specific process choices were made to ensure co-designers felt welcome and to uphold the dignity of co-designers with lived experience [ 34 ]. Examples of process design choices include facilitating transport and parking requests, providing easy access to the building and room, making a sensory breakout room available and having the peer researcher waiting at the entrance to welcome and guide people to the workshop room.

After reaching the workshop room, all co-designers received an individualised resource pack to equalise access to workshop materials, aiming again to balance power in a non-discriminatory way [ 11 ]. The resource pack included name tags with pronouns, individualised refreshments, a fidget toy [ 35 ] whiteboard markers and a human bingo activity described in a later section. An easy-to-apply name tag design was selected after consulting a co-designer with an upper limb difference. Further details on the resource packs are included in Appendix 1 (Supplementary Material 1).

Enabling different kinds of participation

We provided non-verbal response cards to each co-designer as communication preferences vary significantly within the disability community. The cards were intended to benefit any co-designer who struggled to use the response buttons on MS teams. The co-facilitators co-created the Yes, No, and In-the-middle response cards (Fig.  2 ) and were guided by recommendations by Schwartz and Kramer [ 29 ]. They found that people with intellectual disability were more likely to respond “yes” if the negative option included a frowning face or red-coloured images, as choosing these types of alternatives was perceived as being negative or would cause offence [ 29 ].

Non-verbal response cards

A summary of the structure and purpose of each of the five workshops is shown in Fig.  3 , followed by a more in-depth discussion of the strategies employed in each workshop.

Outline of workshop and group structures

Workshop 1: the beginning

Human Bingo was the first workshop activity, as it aimed to support relationship building in an inclusive way for both in-person and online attendees. The activity asked each co-designer to place a name in each worksheet box of someone who fit the described characteristic of that square(for example, someone who likes cooking). To include the two online attendees, laptops were set up with individual videocall streams and noise cancelling headphones enabling the online co-designers to interact one-on-one with others during the activities.

The second activity used The Real Deal cards by Peak Learning [ 36 ] to ask the co-designers to sort cards to prioritise the top five experiences and feelings they would want in a future version of telepractice. This activity aimed to set initial priorities for the redesign of telepractice [ 1 ]. Small groups with a mix of lived experience experts and staff were tasked with negotiating and collaborating to produce their top five desired experiences and feelings for future service success.

A follow-up email was sent after the session to thank co-designers, provide closure, invite feedback and let co-designers know what to expect from the next session.

Workshop 2: mapping the journey

In the second workshop, held online, the co-facilitators explained the journey mapping process and showed a draft of how the visual representation would likely look (Fig.  4 ). As the first step, co-designers were tasked with completing a series of activities to analyse lived experience interview data on the current experience of telepractice for lived experience experts. Small mixed groups were created, prioritising the needs of the lived experience experts to have staff who would be the best fit in supporting them to work through the task [ 1 ]. The small groups were allocated interview quotes corresponding to the steps of a customer journey through telepractice and asked to identify strengths, challenges and emotions associated with the current Telepractice service journey at Rocky Bay [ 1 ]. Further details on the journey map analysis are described in Appendix 1 (Supplementary Material 1) and in a published article co-authored by the co-designers (Benz et al. [ 37 ]).

Draft journey map visualisation

After workshop two, the embedded researcher drafted a journey map by compiling the co-designer group responses to the analysis activity, which was then circulated for feedback and confirmation. The completed journey map is published with further details on the process in an article co-authored with the co-designers, Benz et al. [ 37 ].

Workshop 3: ideas for addressing pain points

For the third workshop, the co-facilitators selected activities to be completed separately by lived experience and staff co-designers. The lived experience expert activity involved exploring preferences for improving pain points identified through the journey map. The lived experience expert activity was facilitated by the peer researcher and support person and included questions such as, how would it be best to learn how to use telepractice? Visual prompt cards were shared to support idea creation, where lived experience expert co-designers could choose any option or suggest an alternative (Fig.  5 ).

Option cards for Lived experience expert co-designer workshop activity

Simultaneously, the staff co-designers completed a parallel activity to address pain points from a service delivery point of view. These pain points were identified in the clinical and non-clinical staff interviews and from the journey map summary of lived experience expert interviews (analysed in Workshop 2). Staff co-designers completed a mind map based on service blueprinting guidelines by Flowers and Miller [ 38 ]. The activity used service blueprinting to identify a list of opportunities for improvement, with four prompts for co-designers to commence planning the actions required to implement these improvements. The foci of the four prompts were roles, policies, technology and value proposition [ 38 ] (described further in Appendix 1 (Supplementary Material 1)). Each of the four prompts were completed for the ten proposed opportunities for improvement to draft plans for future telepractice service delivery.

Workshop 4: story telling and generation of future state solutions

In the fourth workshop, we introduced the concept of prototyping [ 39 ] as a designerly way to test co-designers’ ideas for improving telepractice according to desirability, feasibility and viability with a wider audience of lived experience experts and staff. The co-designers helped to plan the prototyping, and accessibility was a key consideration in selecting a prototype, as the group were conscious of the target audience.

Creating the prototype was collaborative, allowing co-designers to produce an output representing their ideas. They selected a video storyboard prototype with a staff and customer version formatted similarly to a children’s book. It included cartoon animations completed on PowerPoint, voiceover narration, closed captioning and an introductory explanation from two co-designers.

After workshop four, the co-designers collaborated on the customer and staff prototypes during the two weeks between workshops four and five, with support and input from the facilitators. The prototype files were co-produced, with different co-designers working on the visual aspects, the script for the main audio narration and the introductory explanation.

Workshop 5: finishing the story

The co-design group reviewed the draft prototypes in the final workshop, with specific attention paid to the story’s cohesiveness.

The feedback questionnaire was then created to be completed by viewers outside of the co-design group after engaging with either the staff or the customer prototype. The survey allowed Rocky Bay customers and staff to contribute ideas. Following thoughtful discussions, consensus was reached by all co-designers on the final survey questions (Appendix 2 (Supplementary Material 1)).

A reflection activity concluded the final workshop, allowing co-designers to provide feedback on the co-design process, elements for improvement and aspects they valued in participating in the project. Their reflections on the benefits and challenges of co-design in this study are included in the section Co-designer’s perspectives of the workshop series , with the reflection questions included in Appendix 3 (Supplementary Material 1).

Post prototype reflection session

The prototype feedback responses were reviewed with co-designers in a final reflection session. The group then discussed adaptations to the implementation plan for proposal to Rocky Bay. Following the survey discussion, co-designers reviewed proposed service principles for the new telepractice implementation recommendations. These principles aim to align any future decisions in the implementation and service provision stages of the telepractice project with the intentions of the co-designers. An additional reflection activity was completed, specific to the telepractice proposal they had produced and the prototyping process. Feedback relevant to subsequent discussions of the challenges and benefits of co-design is included in the following section: Co-designer’s perspectives of the workshop series , with the reflection prompts in Appendix 3 (Supplementary Material 1).

Benefits and challenges

Learnings derived from completing a study of this kind are complex. However, it is necessary to reflect on which strategies used in the project were beneficial and which strategies created challenges - anticipated and unexpected. These reflections are discussed in two sections, the first being the challenges and benefits reflected upon by co-designers. The second set of reflections relates to organisational and research project-level benefits and challenges from the perspective of clinical department managers and researchers involved in the project.

Co-designer’s perspectives of the workshop series

Co-designers were positive overall about the workshop series. Responses to a prompt for one-word descriptors of their experience included “captivating, innovative, fulfilling, exciting, insightful, helpful, eye-opening and informative ” .

Co-designing as a team

A foundational strategy implemented in this project was the intentional collaboration of lived experience experts with staff; this linked to the co-design principle of prioritising relationships and sharing power. Multiple reflections commented on feeling like a team and that having diverse perspectives across the group was beneficial.

It was especially interesting to hear the perspective of clinicians (for us, the other side of Telepractice). [Lived experience expert Co-designer]

Additionally, the combination of facilitators, including an embedded researcher with an allied health clinical background, a peer researcher with lived experience and a support person with strengths in breaking down tasks, provided different facets of support and task modelling to the co-designers throughout the process.

Balancing idealism and realism

There is an inherent challenge in collaboration between lived experience experts and service providers, whereby co-designers formulate ideas for service improvement and then, in good faith, propose required changes to be implemented. Strategies to support imagination and idealism while being honest about the constraints of what can be delivered were implemented in the context of this project. This was essential to reinforce to co-designers that their contributions and ideas are valid while tempering their hopes with the truth that organisational change is challenging and funding for change is limited. Co-designers were encouraged to be cognisant of ideas that would require high investment (cost and time) and which ideas faced fewer barriers to implementation. This strategy did not prevent the ideation of changes and prioritising what mattered most to them, and co-designers felt it was beneficial in adding a level of consideration regarding what investments they deemed necessary versus those that would be nice to have. For example, having a person to call for help was viewed as necessary, while a nice to have was more advanced technological features.

I feel that the prototype is useful; however, I worry that nothing will be carried over to the Rocky Bay Service. I feel like more customers will want to access telepractice, and Rocky Bay now needs to start the implementation process to ensure that telepractice is utilised, including processes, education and training. [Clinician Co-designer]

The value of small groups

Working in small groups was another beneficial strategy, aiming to create a more hospitable environment for co-designers to voice their thoughts. The small groups varied across activities and workshops, with facilitators intentionally pairing groups that would best support the lived experience of expert co-designers completing activities. As described in the workshop sections, some activities suited mixed groups, whereas others suited lived experience expert and staff-specific groups. Two reflective comments demonstrated the benefit of the small groups, one from a clinician who reflected on supporting a fellow co-designer:

I found that in our group, all of us had a say; however, [Lived Experience Co-designer name] was a bit overwhelmed at times, so I tried to support her with that. [Clinician Co-designer]

And a lived experience expert co-designer additionally reflected:

The breakout rooms were a very good idea. It can be quite intimidating speaking in front of the main group. I found it much easier to participate in the smaller groups . [Lived experience expert Co-designer]

The second session included an unplanned whole group activity, which challenged co-designers. Co-designers reflections of this experience demonstrate the benefits of smaller groups:

I did feel that at the end when the whole group did the task, there wasn’t as much collaboration as there were quite a few more assertive participants, so the quieter ones just sat back. [Clinician Co-designer]

Accessibility and choice

A challenge navigated throughout the workshop series with a diverse group of co-designers was meeting their varying individual health and other needs. This required responding in sensitive, non-judgemental, and supportive ways to encourage co-designers to engage fully. Examples of support include the presence of a support person and adaption of resource packs for co-designers who have difficulty swallowing (re: refreshments), as well as the previously mentioned non-verbal response cards and accessible name tags.

Accessibility supports were also provided for the peer researcher during facilitation activities, including pre-written scripts to provide clarity when explaining tasks to the co-design group, written reminders and regular check-ins. A lived experience expert co-designer reflected that it was beneficial that they could tell the peer researcher was nervous but appreciated that he was brave and made them feel like they did not need to be perfect if the peer researcher was willing to give it a go.

When facilitating the sessions, the embedded researcher and peer researcher identified that the workshops were long and, at times, mentally strenuous. One co-designer requested “more breaks during each session” . Breaks were offered frequently; however, upon reflection, we would schedule regular breaks to remove the need for co-designers to accept the need for a break in front of the group. The instructions for each activity were visual, verbal and written and given at the start of a task. However, once the co-designers were allocated to breakout rooms, they could no longer review the instructions. Many co-designers suggested that having the instructions in each breakout room’s chat window would have been a valuable visual reminder.

One thing I think might of helped a little is having the instructions in the chat as I know I that I listened but couldn’t recall some of the instructions for the group task. [Lived experience expert Co-designer]

Learning new skills and gaining new insight

The co-designers considered that the benefits of working together included learning new skills and widening their understanding of research, the services they provide or use, and the differences between the priorities of lived experience experts and staff. Two lived experience experts commented that the opportunity to learn collaboration skills and create cartoons using PowerPoint were valuable skills for them to utilise in the future. One clinician reflected that the process of co-design had improved their clinical practice and increased their use of telepractice:

My practice is 100% better. I am more confident in using telepractice and more confident that, as a process, it doesn’t reduce the impact of the service- in some ways, it has enhanced it when customers are more relaxed in their own environments. I have not seen my stats, but my use of telepractice has increased significantly, too. [Clinician Co-designer]

The management co-designer acknowledged that although ideas across the group may be similar, prioritisation of their importance can vary dramatically:

Whilst all the feedback and potential improvements were very similar, some things that I viewed as not an issue, was very different to a customer’s perspective. [Management Co-designer]

Overall, the workshop series challenged co-designers. However, the provision of a supportive and accessible environment resulted in mutual benefits for the research, organisation, and co-designers themselves. The strategy for facilitating the workshops was to pose challenges, support the co-designers in rising to meet them, and take into account their capabilities if provided with the right opportunity. A lived experience expert co-designer summarised the effectiveness of this strategy:

I found the activities to be challenging without being too difficult. Each activity provided enough guidance and structure to encourage interesting group discussions and make collaboration easy. [Lived experience expert Co-designer]

Research and organisational reflections of benefits and challenges of co-design

A significant challenge in completing this project was that building foundational relationships and trust takes time. While the authors view this trust as the foundation on which community-based participatory research and co-design are built, they note the direct tension of the time needed to develop these foundational relationships with the timeline expectations of academic and organisational decision-making. The flexibility required to deliver a person-centred research experience for the co-designers resulted in regular instances when timeline extensions were required to prioritise co-designer needs over efficiency. The result of prioritising co-designer needs over research timeline efficiency was an extended timeline that was significantly longer than expected, which sometimes created a disconnect between the flexibility of co-design and the rigidity in traditional academic and organisational processes.

The impacts of a longer-than-expected timeline for completion of the co-design process included financial, project scope, and sponsorship challenges. The project’s initial scope included a co-implementation and co-evaluation phase; however, due to the three-year time constraint, this was modified to conclude following the prototyping process. Whilst the three-year period set expectations for project sponsors and other collaborators from Rocky Bay, the wider context for the project varied significantly and rapidly over this period. This included two changes in Rocky Bay supervisor and one change in Rocky Bay project sponsor. Additionally, one of the academic supervisors left Curtin. This challenge indicates that the project would benefit from key role succession planning.

The peer researcher role was beneficial in providing an opportunity for a person with lived experience to join the study in a strength-based role and experience academic and business processes. However, challenges arose with the timeline extensions, which required this part-time, casual role to be extended by seven months. While the contract extension posed budgetary challenges, the role was viewed as vital to the completion of the project.

While an essential component of research, particularly involving vulnerable populations, ethical approvals proved challenging due to the non-traditional research methods involved in co-design. It was evident to the authors that while the ethics committee staff adhered to their processes, they were bound by a system that did not have adequate flexibility to work with newer research methods, such as co-design. Multiple methods in this study were heavily integrated into the community, including embedded research, peer research and co-design.

The present ethics process provided a comprehensive review focusing on planned interactions within research sessions (e.g. interviews and workshops). Unfortunately, this failed to account for a wider view, including the initial co-production prior to ethical application and anecdotal interactions that occurred regularly in the organic co-design process. In addition to the repeated submissions required to approve the sequential study format, these interactions created a significant workload for the research team and ethics office. These challenges were compounded by the need to navigate Rocky Bay’s organisational processes and changing business needs within ethical approval commitments.

In the authors’ opinion, prioritising the inclusion of lived experience experts in co-creating outputs to disseminate findings was beneficial. The co-creation enabled an authentic representation of the study to audiences regarding community-based participatory research and co-design method implementation. For example, the presentation of a panel discussion at a conference in which the peer researcher could prerecord his responses to questions as his preferred method of participation. All posters presented by the project were formatted to be accessible to lay consumers and were collaboratively produced, with the additional benefit of the posters being displayed across Rocky Bay hubs for customers and staff to gain study insights.

Due to the co-design method’s dynamic nature, some budgetary uncertainty was challenging to navigate. However, financial and non-financial remuneration for all non-staff participants in the project was prioritised. As previously discussed, the position of peer researcher was a paid role; additionally, all lived experience expert participants were remunerated at a rate of AUD 30/hour in the form of gift cards. The carer representative on the steering committee recommended using gift cards to avoid income declaration requirements from government benefits people may receive. Non-financial remuneration for the valuable time and contribution of the co-designer group included co-authorship on an article written regarding the Journey Map they produced (Benz et al. [ 37 ]) and acknowledgement in any other appropriate outputs. The implementation proposal provided to Rocky Bay included recommendations for continued inclusion and remuneration of co-designers.

Setting a new bar for inclusion

Another benefit to reflect upon, which may be the most significant legacy of the project, was setting the precedence for the inclusion of people with disability in decision-making roles in future projects and research conducted by the University and Rocky Bay. After this project commenced, other Rocky Bay clinical projects have similarly elevated the voices of lived experience in planning and conducting subsequent quality improvement initiatives.

I’m lucky enough to have been part of a lot of projects. But I guess I probably haven’t been a part of continuous workshops, pulling in all perspectives of the organisation perfectly… So, collaboration and getting insight from others I haven’t usually was a very unique experience, and I definitely found value if this were to continue in other projects. [Manager Co-designer]

In summary, the findings from using a co-design method for the telepractice research study produced a series of benefits and presented the researchers with multiple challenges. The findings also addressed a literature gap, presenting in-depth descriptive methods to demonstrate how co-design can be applied to a specific case.

Drawn from these findings, the authors identified six main points which form the basis of this discussion. These include (1) the fact that the necessary time and resources required to commit to co-design process completion adequately were underestimated at the outset, (2) there is a need to support the health, well-being and dignity of lived experience expert participants, (3) academic ethical processes have yet to adapt to address more participatory and integrated research methods, (4) strategies used to foster strong collaborative relationships across a diverse group were valued by all participants, (5) better delineation between terminologies such as co-design and community-based participatory research or patient and public involvement would improve the clarity of research methods and author intent and, (6) broader non-traditional impacts that participatory research can create should be better quantified and valued in the context of research impact. Each point will now be discussed in further detail.

In underestimating the time and resources required to complete the telepractice study, a scope reduction was required. This scope reduction removed the study’s originally planned co-implementation and co-evaluation phases. While Harrison et al. [ 40 ] and Bodden and Elliott [ 41 ] advocate for more frequent and comprehensive evaluation of co-designed initiatives, the authors acknowledge that this became no longer feasible within the study constraints. A growing body of literature indicates expected timelines for completed co-production projects from co-planning to co-evaluation. An example by Pearce et al. [ 5 ] indicated that a timeline of five years was reasonable. In contrast, a more limited co-design process was completed with a shorter timeline by Tindall et al. [ 13 ]. Although neither of these articles were published when this study commenced, they are complementary in building an evidence base for future research to anticipate an adequate timeline.

While co-design and other co-production processes are resource and time-intensive, the investment is essential to prioritise the health and other needs of potentially vulnerable population groups in the context of an imbalance of power [ 42 ]. In exploring the concept of dignity for people with disability, Chapman et al. [ 34 ] indicated that recognising the right to make decisions and proactively eliminating or minimising barriers to inclusion are key to protecting dignity. Community participation in decision-making processes such as this study can result in messy and unpredictable outcomes. However, the onus must be placed on policymakers, organisations, and academia to acknowledge this sufficiently rather than demand conformity [ 15 ].

The authors posit that the study would have benefited from an alternative ethics pathway, which may provide additional required flexibility while upholding the rigour of the ethical review process. The increasing frequency of participatory research studies indicates that challenges experienced by the authors of this study are unlikely to be isolated. Lloyd [ 43 ] described challenges regarding information gathered in-between, before and after structured research sessions, reflecting that they relied on personal judgement of the intent to consent for research use. Similarly, Rowley [ 44 ] reflected on the ethical complexities of interacting with families and respecting their confidentiality within the context of being integrated within an organisation. While these studies were co-production in child protection and education, the ethical challenges of their reflections parallel those experienced in the telepractice study. The risks posed by inadequate ethical support in these contexts are that increased poor ethical outcomes will occur, especially in the in-between times of co-design. Therefore, an ethics pathway that involves more frequent brief liaisons with a designated ethics representative to update project progress and troubleshoot ethical considerations may better support researchers to safeguard study participants.

We believe the decision to complete a sequential workshop series with a consistent group of diverse co-designers, led by co-facilitators, was a strength of the co-design process implemented in the telepractice re-design project. The group worked together across a series of workshops, which enabled them to build solid working relationships. Pearce et al. [ 5 ], Rahman et al. [ 16 ] and Tindall et al. [ 13 ] also demonstrated a collaborative whole-team approach to co-design. By contrast, studies that involved separate workshops with different cohorts or multiple of the same workshop did not demonstrate strong collaboration between co-designers [ 18 , 19 , 20 ]. Nesbitt et al. [ 19 ] explicitly highlighted that they would improve their method by completing sequential workshops with a continuous cohort. Stephens et al. [ 45 ] found that small mixed groups were not sufficient to support the participation of people with disability, indicating that the choice to intentionally balance groups to meet the lived experience expert co-designer’s needs may have been an impacting factor on our success.

A lack of clarity in the terminology used in co-design and community-based participatory practice was identified during the completion of this study. We found that co-design frequently meant either a collaborative design process or good participatory practices [ 46 ]. When viewing the structure of the telepractice re-design project, the overarching research approach was community-based participatory-research, and the method was co-design [ 9 ]. The delineation between the overarching approach and methods clarifies the misappropriation of the term co-design with the intent of meaning public participation [ 46 ] rather than the joint process of creative thinking and doing to design an output [ 11 ]. The use of the two-level structure appears more prominent in the United Kingdom, whereas Fox et al. [ 47 ] systematic review assessing public or patient participants identified that 60% of studies originated from the United Kingdom, compared to the next highest 16% for Canada or 4% from Australia and the United States. To improve clarity and reduce confusion about the terminology used, the authors advocate for greater awareness and implementation of the delineation between the concepts of a community-based-participatory-research/patient or public involvement approach versus the co-design method.

An example of co-design being used where alternate terms such as community-based participatory processes (or research) may be more relevant was the most recent amendment to the act governing the NDIS under which this project resided [ 48 ]. The term co-design could be interpreted as an intent to collaborate with people with disability for equitable involvement in all aspects of the NDIS [ 48 ]. It is proposed that the differentiation of these terms would assist in clarifying the intent of the study and dissuade inaccurate expectations of community involvement or design processes.

Implementing community-based participatory research has demonstrated the potential to create an impact that expands further than the original aim of the study. The skills learned by co-designers, the learning of the research team in collaboration with people with disability, the engagement and skill-building of a peer researcher with lived experience, the organisations who engaged in the co-design process and the academic and lay people who engaged with research outputs, all carry a piece of the impact of the co-design process. Rahman et al. [ 16 ] contend that co-design processes positively impact communities. In the context of this study, the peer researcher was included in the National Disability Insurance Agency’s quarterly report as an example of strength-based employment opportunities, which significantly positively impacted his career prospects [ 49 ]. This project provided skills for people with disability that they value and improved the clinical practice of clinician co-designers, which echoes the conclusions of Ramos et al. [ 15 ], who described that participants felt valued and experienced improved self-esteem. There is additional intent from the authors to positively impact disability providers and academia, to advocate for greater collaboration, and to provide open-access publications to provide a stronger evidence base for co-design in clinical practice and service delivery.

Strengths and limitations

The study provides reflective evidence to support the challenges and benefits experienced during the implementation of the study. However, a limitation in the project’s design was the exclusion of outcome measures to assess the impact of process design choices directly. Stephens et al. [ 45 ] completed targeted outcome measures correlating to accessibility adaptations in co-design and conceded that the variability of findings and individual needs reduced the usefulness of these measures.

The reduction of project scope enabled the completion of the study within the limitations of budgeting and timeline restrictions. Although the scope of the project had some flexibility, there were limitations to how far this could be extended as resources were not infinite, and staffing changes meant that organisational priorities changed. Including implementation and evaluation would have improved the study’s rigour. However, Rocky Bay now has the opportunity to implement internally without potential research delays and restrictions.

The blended and flexible approach to the co-design process was a strength of the study as it met the co-designers needs and maximised the project’s potential inclusivity. This strength has the potential to positively impact other studies that can modify some of the process design choices to suit their context and increase inclusivity [ 11 ]. It is believed that the messiness of co-design is important in meeting the needs and context of each individual study; therefore, no two co-design processes should look the same.

The authors concede that the inclusion of a cohort of people with disability and clinical staff does not represent the entirety of their communities, and their proposed changes may cause some parts of the disability community to experience increased barriers [ 50 ]. It is important to note that while the co-designers who participated in this project provided initial design developments, future opportunities remain to iterate the proposed telepractice service and continue to advocate for equitable access for all.

Recommendations for future studies

Recommendations from this study fall into two categories: recommendations for those intending to utilise the described methods and recommendations for future avenues of research inquiry. For those intending to implement the methods, the primary recommendations are to build ample time buffers into the project schedule, implement key role succession planning and set remuneration agreements at the outset, and work together as partners with the mindset that all contributors are creative [ 51 ] with important expertise and invaluable insights if supported appropriately.

Regarding avenues for future inquiry, we recommend investigating a more dynamic and flexible ethics process that may utilise more frequent short consultations to respond to ethical considerations during the emergent co-design and participatory research.

In the authors’ opinion, supported by co-designers experiences, co-design is a useful and outcome-generating methodology that can proactively enable the inclusion of people with disability and service providers in a community-based participatory research approach. The process is both time and resource-intensive; however, in our opinion, the investment is justified through the delivery of direct research benefits and indirect wider community benefits. We advocate for using community-based participatory-research/processes paired with co-design to generate creative thinking within service design processes. Through co-design processes, we recommend collaborating with a single diverse group of co-designers who have the time and space to build trusting working relationships that enable outputs representative of the group consensus.

Data availability

The dataset supporting the conclusions of this article is predominantly included within the article (and its additional files). However, due to the small number of co-designers reflecting upon the research, despite deidentification, there is a reasonable assumption of identification; therefore, the reflection activity response supporting data is not available.

Abbreviations

Australian Dollar

Guidance for Reporting Involvement of Patients and the Public 2 Checklist

Human Research Ethics Committee

Doctor of Philosophy

Patient and Public Involvement

Microsoft Teams

National Disability Insurance Scheme

McKercher KA. Beyond Sticky Notes doing co-design for Real: mindsets, methods, and movements. 1 ed. Sydney, NSW: Beyond Sticky Notes; 2020. p. 225.

Google Scholar  

Mullins RM, Kelly BE, Chiappalone PS, Lewis VJ. No-one has listened to anything I’ve got to say before’: co-design with people who are sleeping rough. Health Expect. 2021;24(3):930–9. https://doi.org/10.1111/hex.13235 .

Article   PubMed   PubMed Central   Google Scholar  

Ekman I, Swedberg K, Taft C, Lindseth A, Norberg A, Brink E, et al. Person-centered Care — Ready for Prime Time. Eur J Cardiovasc Nurs. 2011;4248–51. https://doi.org/10.1016/j.ejcnurse.2011.06.008 . [cited 3/9/2022];10.

National Commission on Safety and Quality in Healthcare. Partnering with Consumers Standard. Australia: National Commission on Safety and Quality in Healthcare. 2021. https://www.safetyandquality.gov.au/standards/nsqhs-standards/partnering-consumers-standard .

Pearce T, Maple M, McKay K, Shakeshaft A, Wayland S. Co-creation of new knowledge: good fortune or good management? Res Involv Engagem. 2022;8(1):1–13. https://doi.org/10.1186/s40900-022-00394-2 .

Article   Google Scholar  

Bordeaux BC, Wiley C, Tandon SD, Horowitz CR, Brown PB, Bass EB. Guidelines for writing manuscripts about community-based participatory research for peer-reviewed journals. Prog Community Health Partnersh. 2007;1(3):281–8. https://doi.org/10.1353/cpr.2007.0018 .

Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. Res Involv Engagem. 2017;3(1):1–11. https://doi.org/10.1186/s40900-017-0062-2 .

Ostrom E, Baugh W, Guarasci R, Parks R, Whitaker G. Community Organization and the Provision of Police Services. Sage; 1973.

Masterson D, Areskoug Josefsson K, Robert G, Nylander E, Kjellström S. Mapping definitions of co-production and co-design in health and social care: a systematic scoping review providing lessons for the future. Health Expect. 2022;25(3):902–13. https://doi.org/10.1111/hex.13470 .

Bibb J. Embedding lived experience in music therapy practice: Towards a future of co-designed, co-produced and co-delivered music therapy programs in Australia. Australian Journal of Music Therapy [Journal Article]. 2022 [cited 2023/08/21];33(2):25–36. https://doi.org/10.3316/informit.829441047529429 .

Davis A, Gwilt I, Wallace N, Langley J. Low-contact Co-design: considering more flexible spatiotemporal models for the co-design workshop. Strategic Des Res J. 2021;14(1):124–37. https://doi.org/10.4013/sdrj.2021.141.11 .

Claborn KR, Creech S, Whittfield Q, Parra-Cardona R, Daugherty A, Benzer J. Ethical by design: engaging the community to co-design a Digital Health Ecosystem to Improve Overdose Prevention efforts among highly vulnerable people who use drugs. Front Digit Health [Original Research]. 2022;4:1–13. https://doi.org/10.3389/fdgth.2022.880849 .

Tindall RM, Ferris M, Townsend M, Boschert G, Moylan S. A first-hand experience of co‐design in mental health service design: opportunities, challenges, and lessons. Int J Ment Health Nurs. 2021;30(6):1693–702. https://doi.org/10.1111/inm.12925 .

Article   PubMed   Google Scholar  

Wahlin DW, Blomkamp DE. Making global local: global methods, local planning, and the importance of genuine community engagement in Australia. Policy Des Pract. 2022;5(4):483–503. https://doi.org/10.1080/25741292.2022.2141489 .

Ramos M, Forcellini FA, Ferreira MGG. Patient-centered healthcare service development: a literature review. Strategic Des Res J. 2021;14(2):423–37. https://doi.org/10.4013/sdrj.2021.142.04 .

Rahman A, Nawaz S, Khan E, Islam S. Nothing about us, without us: is for us. Res Involv Engagem. 2022;8(1):1–10. https://doi.org/10.1186/s40900-022-00372-8 .

Harrison R, Manias E, Ellis L, Mimmo L, Walpola R, Roxas-Harris B, et al. Evaluating clinician experience in value-based health care: the development and validation of the Clinician experience measure (CEM). BMC Health Serv Res. 2022;22:1–11. https://doi.org/10.1186/s12913-022-08900-8 .

Kerr JAS, Whelan M, Zelenko O, Harper-Hill K, Villalba C. Integrated Co-design: a model for co-designing with multiple stakeholder groups from the ‘Fuzzy’ front-end to Beyond Project Delivery. Int J Des. 2022;16(2):1–17. https://doi.org/10.57698/v16i2.06 .

Nesbitt K, Beleigoli A, Du H, Tirimacco R, Clark RA. User experience (UX) design as a co-design methodology: lessons learned during the development of a web-based portal for cardiac rehabilitation. Eur J Cardiovasc Nurs. 2022;21(2):178–83. https://doi.org/10.1093/eurjcn/zvab127 .

Marwaa MN, Guidetti S, Ytterberg C, Kristensen HK. Using experience-based co-design to develop mobile/tablet applications to support a person-centred and empowering stroke rehabilitation. Res Involv Engagem. 2023;9(1):1–17. https://doi.org/10.1186/s40900-023-00472-z .

Tariq S, Grewal EK, Booth R, Nat B, Ka-Caleni T, Larsen M, et al. Lessons learned from a virtual community-based Participatory Research project: prioritizing needs of people who have diabetes and experiences of homelessness to co-design a participatory action project. Res Involv Engagem. 2023;9(1):1–11. https://doi.org/10.1186/s40900-023-00456-z .

Abimbola S, Li C, Mitchell M, Everett M, Casburn K, Crooks P, et al. On the same page: co-designing the logic model of a telehealth service for children in rural and remote Australia. Digit Health. 2019;5:2055207619826468–2055207619826468. https://doi.org/10.1177/2055207619826468 .

Rocky Bay. Rocky Bay Annual Report FY 2021–2022. Perth. 2022. https://www.rockybay.org.au/wp-content/uploads/2022/12/Rocky-Bay-Annual-Report-21-22.pdf .

National Disability Insurance Agency. What is the NDIS? [Internet]. 2021 [updated 14.08.2021. https://www.ndis.gov.au/understanding/what-ndis .

Reen G, Page B, Oikonomou E. Working as an embedded researcher in a healthcare setting: a practical guide for current or prospective embedded researchers. J Eval Clin Pract. 2022;28(1):93–8. https://doi.org/10.1111/jep.13593 .

Bell S, Aggleton P, Gibson A. Peer Research in Health and Social Development 1st Edition ed. London: Routledge; 2021. p. 286.

Book   Google Scholar  

Curran T, Jones M, Ferguson S, Reed M, Lawrence A, Cull N, et al. Disabled young people’s hopes and dreams in a rapidly changing society: a co-production peer research study. Disabil Soc. 2021;36(4):561–78. https://doi.org/10.1080/09687599.2020.1755234 .

Kelly B, Friel S, McShane T, Pinkerton J, Gilligan E. I haven’t read it, I’ve lived it! The benefits and challenges of peer research with young people leaving care. Qualitative Social work: QSW: Res Pract. 2020;19(1):108–24. https://doi.org/10.1177/1473325018800370 .

Schwartz AE, Kramer JM. Inclusive approaches to developing content valid patient-reported outcome measure response scales for youth with intellectual/developmental disabilities. Br J Learn Disabil. 2021;49(1):100–10. https://doi.org/10.1111/bld.12346 .

Webb P, Falls D, Keenan F, Norris B, Owens A, Davidson G, et al. Peer researchers’ experiences of a co-produced research project on supported decision-making. Res Involv Engagem. 2022;8(1):1–10. https://doi.org/10.1186/s40900-022-00406-1 .

People with Disability Australia. PWDA Language Guide: A guide to language about disability. Sydney, Australia. 2021. https://pwd.org.au/wp-content/uploads/2021/12/PWDA-Language-Guide-v2-2021.pdf .

Peters MDJGC, McInerney P, Munn Z, Tricco AC, Khalil H. Chapter 11: Scoping Reviews (2020 version). In: Aromataris E MZ, editor. JBI Manual for Evidence Synthesis, JBI, 2020: JBI; 2020.

Australian Broadcasting Commission. ‘My purpose is changing perceptions’: Australian of the Year Dylan Alcott’s speech in full [Internet]. 2022 [cited 17.08.2023]. https://www.abc.net.au/news/2022-01-26/dylan-alcott-australian-of-the-year-speech-in-full/100783308 .

Chapman K, Dixon A, Ehrlich C, Kendall E. Dignity and the importance of acknowledgement of Personhood for people with disability. Qual Health Res. 2024;34(1–2):141–53. https://doi.org/10.1177/10497323231204562 .

Flattery S. Stim Joy: Using Multi-Sensory Design to Foster Better Understanding of the Autistic Experience: ProQuest Dissertations Publishing; 2023.

Peak Learning. The Real Deal [Internet]. 2023 [cited 6.10.2023]. https://www.peaklearning.com/trd/ .

Benz C, Scott-Jeffs W, Revitt J, Brabon C, Fermanis C, Hawkes M, et al. Co-designing a telepractice journey map with disability customers and clinicians: partnering with users to understand challenges from their perspective. Health Expect. 2023;1–11. https://doi.org/10.1111/hex.13919 .

Flowers E, Miller ME. Your Guide to Blueprinting The Practical Way. 1 ed. USA: Practical By Design 2022. 134 p. pp. 1-134.

Blomkvist J. Benefits of Service Level Prototyping. Des J. 2016;19(4):545–64. https://doi.org/10.1080/14606925.2016.1177292 .

Harrison R, Ní Shé É, Debono D, Chauhan A, Newman B. Creating space for theory when codesigning healthcare interventions. J Eval Clin Pract. 2023;29(4):572–5. https://doi.org/10.1111/jep.13720 .

Bodden S, Elliott J. Finding space for Shared futures. Edinb Archit Res. 2022;37:90–104.

Page K. Ethics and the co-production of knowledge. Public Health Research & Practice. 2022:1–5. https://www.phrp.com.au/issues/june-2022-volume-32-issue-2/ethics-and-co-production/ .

Lloyd J. Life in a lanyard: developing an ethics of embedded research methods in children’s social care. J Children’s Serv. 2021;16(4):318–31. https://doi.org/10.1108/JCS-12-2019-0047 . [cited 2023/12/05];.

Rowley H. Going beyond procedure:engaging with the ethical complexities of being an embedded researcher. Manage Educ. 2014;28(1):19–24. https://doi.org/10.1177/0892020613510119 .

Stephens L, Smith H, Epstein I, Baljko M, McIntosh I, Dadashi N, et al. Accessibility and participatory design: time, power, and facilitation. CoDesign. 2023;1–17. https://doi.org/10.1080/15710882.2023.2214145 .

Gardner G, McKercher KA. But is it co-design? And if it is, so what? 2021. https://healthvoices.org.au/issues/nov-2021/but-is-it-co-design-and-if-it-is-so-what .

Fox G, Lalu MM, Sabloff T, Nicholls SG, Smith M, Stacey D, et al. Recognizing patient partner contributions to health research: a systematic review of reported practices. Res Involv Engagem. 2023;9(1):1–30. https://doi.org/10.1186/s40900-023-00488-5 .

National Disability Insurance Agency. 2022 NDIS legislation amendments Australia; 2022. https://www.ndis.gov.au/news/7975-2022-ndis-legislation-amendments-july-update .

National Disability Insurance Agency. Report to disability ministers for Q4 of Y10 Summary Part A Australia. 2023. https://www.ndis.gov.au/about-us/publications/quarterly-reports .

Lid IM. Universal Design and disability: an interdisciplinary perspective. Disabil Rehabil. 2014;36(16):1344–9. https://doi.org/10.3109/09638288.2014.931472 .

Sanders E, Stappers PJ. Co-creation and the New landscapes of Design. CoDesign. 2008;4:5–18. https://doi.org/10.1080/15710880701875068 .

Download references

Acknowledgements

The authors acknowledge the contribution of Rocky Bay as the industry partner of this project and would like to thank the Co-designers of this project, without whom none of this was possible. The research team would also like to thank Katie Harris for her time and support throughout the workshop series, which were invaluable to the completion of the project and the formation of the published study.

The article forms part of a PhD project funded by the first author, CB’s Australian Government Research Training Program (RTP) scholarship.

Author information

Authors and affiliations.

School of Population Health, Curtin University, Bentley, Australia

Cloe Benz, Richard Norman, Delia Hendrie & Suzanne Robinson

Rocky Bay, Mosman Park, WA, Australia

Will Scott-Jeffs, Mai Welsh & Matthew Locantro

Beyond Sticky Notes, Sydney, Australia

K. A. McKercher

Therapy Focus, Bentley, Australia

Deakin Health Economics, Institute for Health Transformation, Deakin University, Melbourne, Australia

Suzanne Robinson

You can also search for this author in PubMed   Google Scholar

Contributions

CB and MW liaised with the steering committee and conceived the study and structure. SR, DH and RN guided the protocol development and ethics approval. KAM provided methodological support to the project and subject matter expertise. CB and WJS completed participant recruitment, facilitation of workshops and data collection. KAM and CB ideated the format and content of the article. CB completed data analysis and wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved of the final version of the manuscript.

Corresponding author

Correspondence to Cloe Benz .

Ethics declarations

Ethical approval and consent.

The study was approved by the Curtin University Human Research Ethics Committee (ID# HRE2021-0731), and all participants provided written informed consent before engaging in any research activity.

Consent for publication

Not applicable.

Competing interests

Cloe Benz, Richard Norman, Delia Hendrie & Suzanne Robinson do not have any competing interests to declare. Will Scott-Jeffs, Matthew Locantro and Mai Welsh, for all or part of the study period were employed by Rocky Bay a Not-For-Profit Disability Service provider who function as the industry partner for the project. K.A. McKercher is the author of a co-design method book referenced in the article. McKercher also runs a business that helps people co-design.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1:

Appendix 1–3

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Benz, C., Scott-Jeffs, W., McKercher, K.A. et al. Community-based participatory-research through co-design: supporting collaboration from all sides of disability. Res Involv Engagem 10 , 47 (2024). https://doi.org/10.1186/s40900-024-00573-3

Download citation

Received : 13 November 2023

Accepted : 12 April 2024

Published : 10 May 2024

DOI : https://doi.org/10.1186/s40900-024-00573-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Community-based participatory-research
• Telepractice
• Lived experience
• Embedded researcher
• Digital health
• Patient and public involvement

Research Involvement and Engagement

ISSN: 2056-7529

• General enquiries: [email protected]

Green PLM: business goals-oriented algorithm assessing the greenness of a product in the new product development phase for the automotive industry

• Original Research
• Open access
• Published: 13 May 2024

Cite this article

You have full access to this open access article

• Maria Rosienkiewicz   ORCID: orcid.org/0000-0002-6976-352X 1 , 2 ,
• Joanna Helman   ORCID: orcid.org/0000-0002-4239-674X 1 , 2 ,
• Mariusz Cholewa   ORCID: orcid.org/0000-0002-7263-4454 1 , 2 ,
• Mateusz Molasy   ORCID: orcid.org/0000-0001-6390-9711 1 , 2 ,
• Sylwester Oleszek   ORCID: orcid.org/0000-0001-8920-6715 3 &
• Giovanni Berselli   ORCID: orcid.org/0000-0003-0093-3006 4 , 5  

32 Accesses

Explore all metrics

Sustainability-oriented new product development process is becoming of key importance in many sectors of the industry. Especially in the automotive industry, it plays a major role as this sector is historically associated with resource-intensive production and fossil fuel consumption, and thus has to implement solutions that will contribute to restraint of climate change. Manufacturing companies are currently faced with the challenge of adapting their business models to changing market expectations and requirements resulting from the sustainable development path. Moreover, the development of the new digital factories and pressure from the legislators require adjustment of existing Product Lifecycle Management systems. To enhance the concept of the Green Product Lifecycle Management which is still under development, in this work we propose a new three-step methodology composed of three main steps: (1) assessing indicators’ importance for sustainable business goals and business models, (2) assessing indicators and their importance in the automotive industry, (3) assessing “greenness” of the product. This complex approach is innovative as it is based on the business-oriented perspective. Selecting suitable business goals and corresponding business models leads to the selection of the appropriate subset of green indicators. Several more aspects are taken into account in this holistic analysis: the indicator's features, relevance for the company, importance to sustainable business goals, importance in terms of subsectors of the automotive industry as well as environmental, economic, and social impact. This approach enables the company to compute the value of the greenness of the product in reference to the key green indicators.

Avoid common mistakes on your manuscript.

1 Introduction

Sustainability-oriented new product development (NPD) process is becoming of key importance in many sectors of the industry. Especially in the automotive industry, it plays a major role as this sector is historically associated with resource-intensive production and fossil fuel consumption, and thus has to implement solutions that will contribute to restraint of climate change. Embracing sustainability in the new product development process is in fact a strategic necessity for the automotive industry's long-term viability. Practices based on green economy, circular economy, and sustainability not only lead to reducing the net greenhouse gas emissions but also enhancing overall corporate responsibility. When designing a new product, a company is faced with the choice of hundreds, if not thousands, of possible variants ranging from the use of eco-friendly materials to the integration of energy-efficient manufacturing technologies. Selection of the optimal variant of the final product is extremely tough nowadays—it is often a compromise between manufacturing cost, time, and environmental footprint. Therefore, from the scientific point of view, it is very important to support industrial companies with solutions enabling the proper choice in this context. What is more, sustainability-oriented new product development process is in line with the growing consumer demand for environmentally conscious choices, thus fostering brand loyalty and market competitiveness. As the world struggles with the urgent need to transition towards greener economies, the automotive industry's commitment to sustainability not only mitigates environmental, economic, and social impacts but also positions it as a proactive contributor to global efforts in achieving a more sustainable future. Without a doubt, companies operating in the automotive sector are and will be—both directly and indirectly—forced to change their manufacturing processes in order to meet regulations concerning environmental protection. In Europe, a number of documents indicate a direction in which manufacturing companies should be oriented—e.g.:

European green deal which is trying “to make the EU's climate, energy, transport and taxation policies fit for reducing net greenhouse gas emissions by at least 55% by 2030, compared to 1990 levels” (“The European Green Deal” 2021 );

Fit for 55 package which “is a set of proposals to revise and update EU legislation and to put in place new initiatives with the aim of ensuring that EU policies are in line with the climate goals agreed by the Council and the European Parliament” (“Fit for 55” 2023 );

Circular economy action plan which provides “provides a future-oriented agenda for achieving a cleaner and more competitive Europe in co-creation with economic actors, consumers, citizens and civil society organisations. It aims at accelerating the transformational change required by the European Green Deal, while building on circular economy actions” (Commission and Communication 2020 );

Ecodesign directive that successfully regulates energy efficiency and some circularity features of energy-related products ( Directive 2009 /125/EC of the European Parliament and of the Council of 21 October 2009 establishing a framework for the setting of ecodesign requirements for energy-related products (recast) (Text with EEA relevance) 2009 );

EU ecolabel “ helps consumers, retailers and business make truly sustainable choices. The EU Ecolabel logo has become a byword for quality while meeting the highest environmental standards. It means products (goods and services) displaying the iconic "EU flower" symbol meet all the criteria and have earned the right to join the growing EU Ecolabel Community” ( Regulation (EC) No 66/2010 of the European Parliament and of the Council of 25 November 2009 on the EU Ecolabel (Text with EEA relevance) 2009 );

EU green public procurement (GPP) is defined as "a process whereby public authorities seek to procure goods, services and works with a reduced environmental impact throughout their lifecycle when compared to goods, services and works with the same primary function that would otherwise be procured"(“Green Public Procurement” n.d.).

Carbon border adjustment mechanism (CBAM) regulation it is “one of the key elements of the European Union’s ‘Fit for 55’ package.(…) CBAM targets imports of carbon-intensive products, in full compliance with international trade rules, to prevent offsetting the EU’s greenhouse gas emissions reduction efforts through imports of products manufactured in non-EU countries, where climate change policies are less ambitious than in the European Union. It will also help prevent the relocation of the production or the import of carbon-intensive products. The products of the following sectors will be covered by CBAM: cement, aluminium, fertilisers, electric energy production, iron and steel”(“Council agrees on the Carbon Border Adjustment Mechanism (CBAM)” n.d.).

Bearing in mind the above-listed issues, it can be stated that the development of solutions enhancing companies from the automotive industry in the new product development process is of high importance. Therefore, the research presented in this paper focus on this particular topic. On the contrary to well-known methods used widely in the industry (like e.g. Lifecycle Assesment LCA) the authors of this paper propose a new approach in assessing greeness of a product—it is based on a business goals-oriented algorithm applied to the new product development phase for the automotive industry. The presented paper is divided into 5 main chapters. At first literature review is presented and discussed. Literature analysis covers topics related to (1) sustainability-oriented business models, (2) Green Product Lifecycle Management (PLM) and (3) automotive industry. Based on obtained results, the aim of research is defined. Subsequently, a three-step research methodology is described and explained. Finally, results and conclusions are presented. The research—based on which the paper was created—was carried out within the GreenPLM project commissioned by Transition Technologies PSC S.A. This paper is a follow-up of a paper entitled “Towards GreenPLM—Key Sustainable Indicators Selection and Assessment Method Development” (Helman et al., 2023 ). In the mentioned paper the primary scientific goal was to offer a methodical way to choose and assess sustainability indicators for use in the various stages of the automotive lifecycle management. By determining which sustainability indicators are pertinent to each stage of the product lifecycle, the application's principal objective is to assist the industry in its pursuit of more environmentally friendly development. As a result, the important green indicators for the automotive sector in line with the GreenPLM concept are highlighted, together with their classification into the early stages of the car's lifecycle and prospective data sources. The paper written by Helman et al. introduces the idea of GreenPLM and the potential uses for it in the future (Helman et al., 2023 ). In the present paper, further analysis is carried out, which focuses on the development of a new—based on sustainability- oriented business goals—approach enabling assessing greenness of a product in the automotive industry.

2 Literature analysis

2.1 sustainability-oriented business models.

Nowadays, a highly developed economy and strong market competitiveness force companies not only to continuously increase the effectiveness in every area, but also—like never before—to take into consideration environmental aspects (Kochańska, 2024 ). Manufacturing companies are currently faced with the challenge of adapting their business models to changing market expectations and requirements resulting from the sustainable development path. Business model can be defined as “a conceptual tool containing a set of objects, concepts and their relationships with the objective to express the business logic of a specific firm” (Osterwalder et al., 2005 ). It helps to understand how a company does a business and it can be used for analysis, comparison and performance assessment, management, communication, and innovation (Bocken et al., 2014 ). It answers a series of questions essential to any business, i.e. who are the customers, what do they value, how that value can be delivered to the customer at an appropriate cost and how the business deploys its assets. Business model includes a description of the key assets and are concerned with how the firm defines its competitive strategy through the design of the product or service it offers to its market, how it charges for it, what it costs to produce, how it differentiates itself from other firms by the value proposition, and how the firm integrates its own value chain with those of other firm’s in a value network (Rasmussen, 2007 ). Business models cover two main spheres: the first sphere is operations and related resources, the second sphere is about created and captured value (Rudnicka, 2016 ). The central point of every business model is value proposition (“BS8001:, 2017 BSI Knowledge” 2017 ). As presented in the Introduction, changing business models into those supporting green economy, circular economy and sustainability is no longer a need, but rather necessity. Business models (innovations) for sustainability can be defined as “innovations that create significant positive and/or significantly reduced negative impacts for the environment and/or society, through changes in the way the organization and its value-network create, deliver value and capture value (i.e. create economic value) or change their value propositions” (Bocken et al., 2014 ). Also, the transition to a circular economy requires significant changes in the way organizations operate. The transition to the circular economy often entails holistic adaptations in firms' business models or even the creation of new ones. The implementation of circular economy principles often requires new visions and strategies and a fundamental redesign of product concepts, service offerings, and channels towards long-life solutions. CE principles also affect how companies can make money, with the ownership structure perhaps shifting, boosting demand for services along the product lifecycle (Ferasso et al., 2020 ). Early developments of sustainable business models link the concept of sustainability with innovation. Namely, the sustained success of an organization depends on innovation. Rules determining the functioning of a sustainable business model need to be based on creating technological innovations that can create new markets after being commercialized (Boons & Lüdeke-Freund, 2013 ). Schaltegger et al. define a business model for sustainability as one which helps in describing, analyzing, managing, and communicating as follows (Schaltegger et al., 2016 ):

a company’s sustainable value proposition to its customers, and all other stakeholders,

how it creates and delivers this value,

how it captures economic value while maintaining or regenerating natural, social, and economic capital beyond its organizational boundaries.

Wells states that “a sustainable product cannot be produced by an unsustainable industry” (Wells, 2013 ). The author considers defining a sustainable automotive industry that is “one that creates life-enhancing employment for communities over a long period of time. It has zero net consumption of physical resources in production. It is consistently profitable while being able to withstand short-term fluctuations in economic circumstances. And it produces products that themselves do not pollute or otherwise degrade the environment, are fit for purpose, and are designed for longevity” (Wells, 2013 ). This definition suggests that, over time, manufacturing as such (of new, complete products) is only a small part of the business model, that concepts such as product-service systems are more appropriate. It is recognised that profitability is absolutely vital for sustainability. However, profitability is a necessary but insufficient condition for sustainability: the environmental and social dimensions must also be included.

Jonker and O’Riordan define four criteria that a sustainable business model must meet. They are: sharing knowledge, making connections (sustainability is between companies), awareness and multiple value creation (Jonker & O’Riordan, 2016 ). As the traditional business model elements are insufficient for showing the full relationships and values into the direction of sustainable development in business context, Rudnicka presented elements of business models including sustainability issues in (Rudnicka, 2016 ). Bocken et al. identified eight sustainable business model archetypes on the basis of analysis of practice and literature business model examples. The archetypes are classified in higher order groupings that describe main types of business models innovations: technological, social, and organizational oriented innovations. They are as follows (Bocken et al., 2014 ):

maximizing material and energy efficiency,

creating value from ‘waste’,

substitute with renewables and natural processes,

delivering functionality rather than ownership,

adopting a stewardship role,

encouraging sufficiency,

re-purposing the business for society/environment,

developing scale-up solutions.

Clinton and Whisnart presented 20 business model innovations for sustainability. They have been categorized into 5 categories, however authors admit that some business models fit into multiple categories. They are as follows (Clinton & Whisnant, 2019 ):

category of environmental impact

Closed Loop Production,

Physical to Virtual,

Produce on Demand,

Rematerialization,

category of social impact

Buy One, Give One,

Cooperative Ownership,

Inclusive Sourcing,

category of financial impact

Crowdfunding,

Differential Pricing,

Innovative product financing,

Pay for Success,

category of social innovation focused on “the base of the pyramid”

Building a Marketplace,

Microfinance,

Micro-Franchise,

Subscription Model,

category of diverse impact

Alternative Marketplaces,

Behavior Change,

Product as a Service,

Shared Resource.

The value proposition seems central to any business model, because it defines the products/services that create value for the organization’s existing and/or new customer base. It also relates to the viability of the business model, because it identifies how the organization makes money. For organizations transitioning to more circular and sustainable operations, the value proposition might not be solely financial in nature and instead reflect delivery of wider social and/or environmental benefits (“BS8001:, 2017 BSI Knowledge” 2017 ). Referring to the circular economy, Mentink ( 2014 ) identified the changes of business model components needed for developing a more circular service model. They are as follows:

“value propositions (what?)—products should become fully reused or recycled, which requires reverse logistics systems, or firms should turn towards product-service system (PSS) and sell performance related to serviced products‚

activities, processes, resources and capabilities (how?)—products have to be made in specific processes, with recycled materials and specific resources, which may require not only specific capabilities but also creating reverse logistics systems and maintaining relationships with other companies and customers to assure closing of material loops,

revenue models (why?)—selling product-based services charged according to their use‚

customers or customer interfaces (who?)—selling circular products or services may require prior changes of customer habits or, if this is not possible, even changes of customers” (Lewandowski, 2016 ).

Laubscher and Marinelli (Laubscher & Marinelli, 2014 ) identified six key areas for integration of the circular economy principles with the business model. They are as follows: sales mode, product design/material composition, IT/data management, supply loops, strategic sourcing for own operations, HR/incentives (Lewandowski, 2016 ).

In order to provide a framework for and guidance on implementing the principles of the circular economy within organizations, the British norm BS 8001:2017 has been developed (“BS8001:, 2017 BSI Knowledge” 2017 ). The guidance is intended to apply to any organization, regardless of location, size, sector and type. In the norm six general business model groupings have been introduced that have the potential to be compatible with a circular economic system. They are as follows and they include following specific business models (“BS8001:, 2017 BSI Knowledge” 2017 ):

business model “produce on demand/made to order”,

Dematerialization

business model “digitalization”,

Product lifecycle extension/reuse

business model “product life‑extension”,

business model “facilitated reuse”,

business model “product modular design”,

business model “refurbish, repair, remanufacture and recondition”.

Recovery of secondary raw materials/by‑products

business model “recovery of secondary materials/by‑products (including recycling)”,

business model “incentivized return/extended producer responsibility”.

product as a service/product–service system (PSS)

business model “lease agreement”,

business model “performance based (pay for success)”,

sharing economy and collaborative consumption

business model “sharing economy”,

business model “sharing platforms/resources (collaborative consumption)”.

It has been assumed that these business models are not mutually exclusive. In order to realize its value proposition an organization might need to adopt a business model which includes elements of each of them. Moreover, complex organizations might concurrently have different business models operating across their divisions/units and for different product/markets (“BS8001:, 2017 BSI Knowledge” 2017 ).Popular circular economy business models collected by Lewandowski are as follows (Lewandowski, 2016 ):

classification criteria: renegade

Energy recover—the conversion of non-recyclable waste materials into useable heat, electricity, or fuel,

Circular supplies—using renewable energy,

Efficient buildings—locating business activities in efficient buildings,

Sustainable product locations—locating business in eco-industrial parks,

Chemical leasing—the producer mainly sells the functions performed by the chemical, so the environmental impacts and use of hazardous chemical are reduced,

classification criteria: share

Maintenance and Repair—product lifecycle is extended through maintenance and repair,

Collaborative Consumption, Sharing Platforms, PSS Footnote 1 : Product renting, sharing or pooling—enable sharing use, access, or ownership of product between members of the public or between businesses,

PSS: Product lease—exclusive use of a product without being the owner,

PSS: Availability based—the product or service is available for the customer for a specific period of time,

PSS: Performance based—the revenue is generated according to delivered solution, effect or demand-fulfillment,

Incentivized return and reuse or Next Life Sales—customers return used products for an agreed value. Collected products are resold or refurbished and sold,

Upgrading—replacing modules or components with better quality ones,

Product Attachment and Trust—creating products that will be loved, liked or trusted longer,

Bring your own device—users bring their own devices to get the access to services,

Hybrid model—a durable product contains short-lived consumables,

Gap-exploiter model—exploits “lifetime value gaps” or leftover value in product systems. (e.g., shoes lasting longer than their soles),

classification criteria: optimize

Asset management—internal collection, reuse, refurbishing and resale of used products,

Produce on demand—producing when demand is present and products were ordered,

Waste reduction, Good housekeeping, Lean thinking, Fit thinking—waste reduction in the production process and before,

PSS: Activity management/outsourcing—more efficient use of capital goods, materials, human resources through outsourcing,

classification criteria: loop

Remanufacture, Product Transformation—restoring a product or its components to “as new” quality,

Recycling, Recycling 2.0, Resource Recovery—recovering resources out of disposed products or by-products,

Upcycling—materials are reused and their value is upgraded,

Circular Supplies—using supplies from material loops, bio based- or fully recyclable,

classification criteria: virtualize

Dematerialized services—shifting physical products, services or processes to virtual,

classification criteria: exchange

New technology—new technology of production.

On the basis of analysis of fore mentioned business models Lewandowski ( 2016 ) proposes a circular business model canvas, that is extended and adjusted to the circular economy version of the business model canvas developed by Osterwalder and Pigneur ( 2010 ). The building blocks allow for designing of a business model according to the principles of circular economy, and consists of value propositions, customer segments, channels, customer relationships, revenue streams, key resources, key activities, key partnerships, cost structure, take-back system, adoption factors (Lewandowski, 2016 ). Another nineteen circular business models grouped into 6 groups have been proposed by IMSA, an independent think tank and consultancy & research firm committed to the environment, sustainability and innovation, headquartered in Amsterdam. They can be found in (“Circular Future—Public Affairs and consultancy about Circular Economy” 2022 ). Four main business models that circulate products and materials in the economy, and have the potential to decouple revenue streams from production and resource use, have been presented by Ellen MacArthur Foundation. They are: “Resale” business model, “Rental” business model, “Repair” business model, “Remaking” business model (Ellen MacArthur Foundation, 2021 ). The business model approach seems useful for analyzing and improving sustainability, because business models can be viewed as activity systems that cover internal and external activities in which companies are engaged. When activities represent sustainable issues, they are best represented by the practical descriptions, communication, and expression of activities of companies (Ritala et al., 2018 ). Analysis of Green Supply Chain Management operations used by Ford, BMW, Toyota, and BYD has been performed by Sanghavi et. al and is presented in (Sanghavi et al., 2015 ).

Literature review focused on sustainability-oriented business models shows that a lot of studies have been dedicated to this topic. Numerous classifications have been developed and described. Companies, depending on their business goals, can suitably choose a proper business model to start their transition process towards sustainability and green economy.

2.2 Green product lifecycle management

Product Lifecycle Management is facing a major change. New phenomena are emerging in new product development—including crowdsourcing, crowdfunding, and gamification (Molasy et al., 2023 ). Development of the new digital factories and pressure from the legislators require adjustment of existing PLM systems. They need to support the companies in terms of collaborative processes and workflows to enhance sustainable information sharing.

According to Belkadi et al., the key challenge in this context is not only the definition of information flow between all involved entities in the sustainability process but also the mapping of information between different production lifecycle phases (Belkadi et al., 2015 ). In the investigated area a number of definitions have been developed to better precise the emerging area, including sustainable products (Bangsa & Schlegelmilch, 2020 ), Closed Loop Lifecycle Management (CL2M), Green Information Systems, Sustainable PLM, and Intelligent Products (Främling et al., 2013 ). In terms of sustainable products a number of indicators can be considered in order to provide the customers with the appropriate assurance and information about their sustainability performance (Bangsa & Schlegelmilch, 2020 ). According to Främling et al. Sustainable PLM can be defined as a type of Closed Loop Lifecycle Management (CL2M). The key feature is the possibility to connect Intelligent Products with each other and at the same time with other information systems. The systems that are called Green Information Systems (Green IS) aim at reducing environmental impacts (Främling et al., 2013 ). Their focus is to reduce”the production of CO 2 and other greenhouse gases through new combinations of people, processes, and technologies that enable the processing of digitized information” (Främling et al., 2013 ). Främling et al. underline that the main purpose of CL2M is to constantly improve design, manufacturing, use and end-of-life handling of products, in order to obtain "improved quality, less breakdowns, reduced need for spare parts and ensuring an operation that is continuously maintained at the most energy- and resource-efficient level” (Främling et al., 2013 ). According to Främling et al. Sustainable PLM should be understood as CL2M for the purpose of improving environmental sustainability during all phases of the lifecycle. A key issue in terms of Sustainable PLM or so called Green PLM is related to measurement of environmental sustainability and ways to improve it. Therefore, it is crucial to measure, gather, and analyse data and corresponding indicators which can be a firm basis and a trigger for companies in their shift towards green production. A helpful approach in this process is LCA, which stands for Lifecycle Assessment or Lifecycle Analysis defined a methodological framework for estimating and assessing the environmental impacts attributable to the lifecycle of a product (Främling et al., 2013 ). According to Främling et al., it is hard to extend and/or adapt existing ERP systems to the requirements of Sustainable PLM. Främling et al. state that “it is likely that an easier, safer, and more scalable approach is to develop and increase the use of Intelligent Product-based information systems separately and communicate with stock-based information systems using their own external interfaces. The multi-organizational, multi-manufacturer, etc., reality and requirements of Sustainable PLM can only be met in a scalable way if ad hoc, loosely coupled interoperability between devices and other information systems is technically easy and economically feasible enough. Failing to recognize the importance of loosely coupled integration can delay the speed of evolution and adoption of Sustainable PLM” (Främling et al., 2013 ). According to Vila et al. ( 2015 ) a Green PLM strategy should be composed of mission, vision and objective. They propose an approach to product lifecycle with green competencies as well as Green Product Lifecycle Framework. Green Product Lifecycle Framework should be composed of 3 main phases (Vila et al., 2015 ):

Design-Development phase, focused on Eco Design and Green Development, composed of: Strategic Planning, Conceptual Design, Embodiment Design, Detail Design, Manufacturing Plan;

Manufacturing phase focused on Green Manufacturing and Sustainable Production, composed of: Storage Package, Assembly, Production, Production Control, Resource Management;

Service phase, focused on Sustainable Logistics, Product Special Response and Responsible Use and Maintenance, composed of: Logistics, Sales, Delivery, Client Service, Reduce/Reuse/Retire/Recycle;

For each of the 3 above-mentioned phases, Vila et al. propose an approach based on dedicated methods, tools and knowledge, as follows (Vila et al., 2015 ):

Design-Development phase:

Methods : Design for environment (Eco Design), Eco-standards application

Tools : Eco CAD/CAE/CAM tools, Lifecycle management tools (PLM)

Knowledge : Product lifecycle assessment, Materials and process selection

Manufacturing phase:

Methods : Design for sustainable manufacturing, Green Supply chain management

Tools : Green manufacturing, Intelligent/smart sensors/controls, Green energies

Knowledge : New materials, New manufacturing technologies

Service phase:

Methods : Design for Sustainable Maintenance, Sustainable service system

Tools : Product Usefulness Time Control, Sustainable Use Scorecard

Knowledge : Energy consumption (CO2 impact), Maintenance and care.

Marcon et al. propose another framework for a green product. They divided green product attributes' groups into 3 main product lifecycle phases, namely—(1) Production, (2) Use and (3) End-of-life (Marcon et al., 2022 ). Recently, the Brazilian automotive industry has been pressured by stricter environmental laws and regulations focusing on adopting more green production processes, whereas consumers have increased demand for green products. In this context, De Medeiros et al. investigated behavioural motivation for green product consumption. They focused their research on the decision-making process regarding the purchase of cars and furniture (de Medeiros et al., 2016 ). According to the obtained results de Medeiros et al. prove that “if the prices of products with and without ecological appeal were the same, 95% of the respondents would prefer to buy the green product”. This key finding from this research provides the manufacturers with very important information.

2.3 Automotive industry insight

The automotive industry comprises of a wide range of companies and organizations involved in the design, development, manufacturing, marketing, and selling of motor vehicles. It can be divided into sectors in many different aspects. For the purpose of this research the four classifications were taken into account—namely: (1) Global Industry Classification Standard, (2) Industry classification taxonomy, (3) Refinitiv Business Classification and (4) Automotive supply chain actors.

The Global Industry Classification Standard, developed by S&P Dow Jones Indices, was designed in response to the global financial community’s need for accurate, complete and standard industry definitions (“Global Industry Classification Standard” 2018 ). The GICS structure consists of 11 Sectors, 24 Industry groups, 69 Industries and 158 sub-industries. Under the sector of Consumer Discretionary, the Automobiles & Components industry group can be found. It is divided into two industries with two sub-industries (each) are indicated. They are as follows:

Auto Components

Auto Parts and Equipment

Tires and Rubber

Automobiles

Automobile Manufacturers

Motorcycle Manufacturers

Industry classification taxonomy developed for the Industrial Classification Benchmark (ICB) is a widely used standard of detailed and comprehensive structure for sector and industry analysis, facilitating the comparison of companies across four levels of classification and national boundaries. Its taxonomy covers a robust system of 11 Industries with 20 Supersectors and 45 Sectors detailed into 173 Subsectors (“Industry Classification Benchmark” 2022 ). Under the industry of Consumer Discretionary, the supersector of Automobiles and Part can be found. It is divided into 4 subsectors (“Industry Classification Benchmark” 2022 ):

Auto Services—Companies that provide assistance to individual vehicle owners.

Tires—Manufacturers, distributors and retreaders of automobile, truck and motorcycle tires

Automobiles—Makers of passenger vehicles, including cars, sport utility vehicles (SUVs) and light trucks. Excludes makers of heavy trucks and makers of recreational vehicles (RVs and ATVs).

Auto Parts—Manufacturers and distributors of new and replacement parts for motorcycles and automobiles, such as engines, carburettors and batteries. Excludes producers of tires, which are classified under Tires Subsector.

A similar division is presented by Refinitiv Business Classification (previously called Reuters Business Sector Scheme as well as Thomson Reuters Business Classification, developed by the Reuters Group. It is a market-based classification scheme, similar to the GICS and ICB systems. The automotive industry is represented by Consumer Cyclicals economic sectors in Automobiles & Auto Parts business sector. The industry group of Automobiles & Auto Parts covers 3 types of industries that are divided into the following activities (“TRBC Sector Classification” n.d.):

Auto and Truck Manufacturers

Motorcycles and Scooters

Automobiles and Multi Utility Vehicles

Light Trucks

Electric (Alternative) Vehicles

Luxury Vehicles

Auto & Truck Wholesale

Auto, Truck & Motorcycle Parts

Automotive Body Parts

Engine & Powertrain Systems

Automotive Batteries

Automotive Systems

Automotive Accessories

Motorcycle Parts & Accessories

Auto & Truck Parts Wholesale

Tires & Rubber Products

Tire & Tube Manufacturers

Tire Retreading

Industrial Rubber Products

Rubber Plantations

Tires & Rubber Products Wholesale

The other type of the classification of the automotive industry is its division into supply chain actors, where at least 4 levels can be distinguished (“Driving Innovation for Automotive Manufacturing Tier 1 Suppliers” n.d.):

Tier 3—suppliers of raw and semi-raw materials,

Tier 2—non-automotive grade parts,

Tier 1—automotive parts and systems,

OEM—Original Equipment Manufacturer.

Above-presented classifications prove that the automotive industry is complex and is divided into many subsectors. Its influence from the global perspective is huge. Thus, the car sector is regarded as one of the primary causes of the environmental problem on the planet. Additionally, it has been demonstrated that the social and environmental aspects are negatively impacted by existing methods in the automotive industry. On the other hand, any nation's economic development depends heavily on the automotive industry. As a result, the sector needs to drastically alter its current operational procedures (Helman et al., 2023 ). Currently, according to Golinska-Dawson ( 2019 ) the Automotive Parts Remanufacturers Association (APRA) “identifies over 50 different components which are currently remanufactured”. Although the material flow in remanufacturing process can be assessed as much more complex than in the primary manufacturing process (Golinska-Dawson & Pawlewski, 2015 ), Kosacka-Olejnik et al. ( 2020 ) emphasise that remanufacturing is set to become an increasingly important player in the future manufacturing industry with an EU market potential of €90 billion by 2030 according to the European Remanufacturing Network. Certain publications mention that sustainability-oriented goals are still not seen as being as significant in the manufacturing sector as traditional productivity targets, yet this is slowly changing (Kochańska & Burduk, 2023 ). Sliż and Wycinka underline that in order to decrease the negative influence on the economy and environment, attempts are made to design vehicles in a way which reduces both CO 2 emissions, and fuel consumption as well as replaces internal combustion engines with electric units (Sliż & Wycinka, 2021 ). To reduce negative environmental impact, engine emission standards are getting stricter – i.a. ambitious goals to reduce the concentrations of hazardous compounds in the exhaust gases were supplemented by strict requirements for the reduction of the concentration and volume of solid particles (Žvirblis et al., 2023 ). It is crucial to develop methods that allow to perform the same processes but with the more ecologically-oriented solutions—i.e. by the reduction of pollutant emissions (Łapczyńska, 2023 ) or by optimizing transportation processes, as they are one of the most important sources of waste, no matter how necessary transport is. (Kochańska et al. Early Access). All the mentioned requirements and trends cause the automotive industry to make the new product development process greener. Therefore, in the automotive sector, sustainability metrics, particularly those pertaining to the environmental aspect, have gained substantial attention.

2.4 Significance and aim

The literature review presented in the previous chapters leads to certain important conclusions. First of all, many business models for sustainability have been developed so far. Companies, depending on their business goals, can suitably choose a proper business model to start their transition process towards sustainability and a green economy. Moreover, the transition towards digital manufacturing and legislative pressure cause a need to modify the current PLM systems. They must assist the businesses with their collaborative workflows and processes in order to improve sustainability- and green-oriented processes. The concept of the Green Product Lifecycle Management is still under development. It can also be noted that the new product development (NPD) phase is crucial from the perspective of further environmental impact. In general, there are certain methods—the majority of them based on Lifecycle Assessment (LCA) approach, yet a dedicated solution based on sustainability-oriented business models is missing. Last, but not least, it should be emphasized that especially the automotive industry—due to its huge impact on the environment—requires a radical shift in the way it performs usual business practices Based on these conclusions, the main aim of the paper was derived. Bearing in mind a number of existing sustainability-oriented business models, and a need of development PLM systems towards greenness (especially in the NPD process) and major shift within the automotive industry, the main aim of this paper is to develop a new approach to assessing greenness of a product which would be based on a business goals-oriented algorithm and applied to the new product development phase for the automotive industry. This algorithm will be applied to the 20 indicators identified within the first stage of the research described in the paper “Towards GreenPLM—Key Sustainable Indicators Selection and Assessment Method Development” (Helman et al., 2023 ).

3 Research methodology

In order to reach the above-mentioned aim, the three-step research methodology was proposed. It is composed of 3 main steps:

Step 1 : assessing indicators’ importance for sustainable business goals and business models.

Step 2 : assessing indicators and their importance in the automotive industry.

Step 3 : assessing “greenness” of the product.

To assess the 20 indicators (selected in (Helman et al., 2023 )) in terms of their importance for sustainable business goals and business models, the step 1 of the research methodology—presented in Fig.  1 —has been developed. In the first stage, business goals need to be analyzed with respect to sustainability issues. Parallelly, business models focused on sustainability aspects should be analyzed. Next, to each of the defined business goals, corresponding business models must be identified, and their impacts on sustainability should be indicated (selecting from: environmental, economic and/or social). The final step of the methodology is to assign indicators that can be used to achieve defined business goals and corresponding business models.

Step 1: Methodology of assessing indicators’ importance for sustainable business goals and business models; own elaboration

Once the 20 indicators will be assessed in terms of their importance for sustainable business goals and business models according to the step 1, the methodology for assessing indicators and their importance in the automotive industry will be applied (step 2). The scheme of the proposed methodology is presented in the figure below. At first five overarching assessment criteria are defined, namely:

Indicator's features,

Relevance for the company,

Importance to sustainable business goals,

Importance in terms of subsectors,

Environmental, Economic, Social impact.

Some of the overarching criteria have been detailed to more precisely assess the indicators:

Indicator's features:

availability of data,

durability (low variability over time),

need of monitor—cost of collecting,

preciseness/quality/confidence level,

ease of indicator-related data processing,

standardized indicator,

Importance in terms of subsectors:

Auto Services,

Automobiles,

Auto Parts,

Environmental, Economic, Social impact:

Environmental impact:

Raw material extraction,

Modification of natural area,

Economic impact:

Material cost of product,

Maintenance cost of product,

Manufacturing cost of product,

End-of-life cost of product,

Social impact:

Health and safety,

Socio-economic repercussion,

Governance by customers,

Human rights.

According to the step 2 of the research methodology presented in Fig.  2 , each assessment criterion has to be analyzed and rated. The ratings for particular criteria are presented in the figure above. Once the points are assigned, an overall criterion assessment has to computed—for each overarching criterion separately. Subsequently, all calculated values have to be normalized to enable mutual comparison and further analysis. Thus mean of the overall normalized assessment for all five criteria can be estimated. Finally, the indicators can be sorted from the most relevant to the least relevant and the key sustainability-oriented indicators for Green PLM can be identified.

Step 2: Methodology for assessing importance of indicators for the Green PLM project

Once the key indicators are selected according to the step 2 of the methodology of assessing indicators and their importance in the automotive industry, they can be further analyzed in order to indicate how to use them in terms of the “greenness” of the product. The algorithm presented in Fig.  3 can serve as a tool helping a company to assess to what extent their product can be defined as “green”.

Step 3: Algorithm assessing the “greenness” of a product

A scheme presenting the step 3 of the research methodology, which is an algorithm assessing the “greenness” of a product, is in the figure below. It is composed of 10 steps.

In order to initiate the algorithm assessing the “greenness” of a product, a company has to first determine its business goals. Next, based on that, relevant indicators should be selected. In the following step, values of the indicators should be defined and (if needed) results aggregated. Next, the corresponding values of the final product and the critical value of the indicators in terms of the greenness of the product need to be defined. Subsequently, results for all analyzed indicators should be calculated. If the results meet the criterion 1 point for each indicator is assigned. Within another step, the weight of each of the criteria has to be defined and then summed up (value X). Next, for each indicator, the point value has to be multiplied by the criterion's weight and then summed up (value Y). Finally, Y by X should be divided. The obtained value defines the level of greenness of the product in percentage terms.

4.1 Step 1: assessing indicators’ importance for sustainable business goals

In the following chapters, the results obtained from the described three-step research methodology will be presented. At first, the procedure of assessing green indicators’ importance for sustainable business goals will be applied.

According to the step 1 of the research methodology, business goals and business goals have to be identified at first. The review of numerous sustainability-oriented business models is presented in chapter 2.1, enabled to define the final list of the business models used in the further analysis. The most suitable business models that are reflecting sustainable business goals were identified by:

British norm BS 8001:2025 (“BS8001:, 2017 BSI Knowledge” 2017 ),

Lewandowski ( 2016 ),

IMSA (“Circular Future—Public Affairs and consultancy about Circular Economy” 2022 ),

Clinton & Whisnart ( 2019 ),

Ellen MacArthur Foundation ( 2021 ).

They are as follows:

Produce on demand/made to order,

Produce on demand,

Remanufacture, Product Transformation,

Recycling, Recycling 2.0, Resource Recovery,

Closed loop production,

Upcycle/upcycling,

Digitalization,

Dematerialized services,

Physical to virtual,

Product life‑extension,

Product as a service,

Circular supplies,

Facilitated reuse,

Product modular design,

Next life sales,

Refurbish and resell,

Refurbish, repair, remanufacture and recondition,

Recovery of secondary materials/by‑products (including recycling),

Recycling (waste handling and repurpose),

Incentivized return/extended producer responsibility,

Take back management,

Asset management,

Incentivized return and reuse or next life sales,

Lease agreement,

Pss: product lease,

Sharing Economy,

Collaborative Consumption, sharing platforms, pss: product renting, sharing or pooling,

Collaborative production,

Shared resource,

Sharing platforms/resources (collaborative consumption),

Sharing platforms,

Pay per use,

Buy one, give one,

Inclusive sourcing,

Maintenance and repair,

Behavior change,

Energy recover,

Waste reduction, good housekeeping, lean thinking, fit thinking,

Chemical leasing.

The business goals focused on implementing the principles of green economy are defined as follows:

to minimize raw material and/or energy demand,

to avoid overstocking,

to replace physical infrastructure and assets with digital/virtual services,

to design new products to be durable for a long lifetime (durability),

to reuse parts/components with or without repair/upgrade,

to encourage cost effective product repairs and reduce need for replacement of integrated components, thereby reducing resource consumption,

to put the product back into the market to earn a second or subsequent income,

to create products from secondary raw materials/by products and recycling,

to recover used/unwanted products and sell it again,

to increase overall profitability during the lease period,

to strengthen community relationships,

to reduce costs over directly sourcing the products/services,

to increase utilization rate of products and services,

to create an entirely new product from the source material (waste),

to attract consumers to the cause (to donate a portion of the profits to those in need),

to create more ‘ethical’ products by setting supplier standards or conducting audits (toward sustainable sourcing),

to extend product lifecycle,

to increase “stickiness” with the customer, making them less likely to buy from another good/service provider,

to convert non-recyclable waste materials into useable heat, electricity, or fuel,

to reduce waste in the production process and before,

to cooperate in the production value chain leading to closing material loops,

to reduce environmental impacts and use of hazardous chemical.

Below, for the three selected business goals, the examples of the explanatory results are presented.

Business goal: to minimize raw material and/or energy demand (Table  1 ).

Corresponding indicators:

lightweight material

sustainable material

number of renewable parts

number of reused parts assembled

energy usage

water usage

amount of waste

product size (partly)

product weight (partly)

amount of scrap (partly)

Business goal: to reuse parts/components with or without repair/upgrade (Table  2 ).

energy usage water usage

waste recycled/reused

Business goal: to extend product lifecycle (Table  3 ).

number of reused parts assembled (partly)

Analogous analysis was performed for all the identified business goals. In the next step all indicators were analyzed under the aspect of their suitability to reach sustainable business goals. Indicators were assigned as corresponding (blue color in Table  4 ) or partly corresponding (light blue color in Table  4 ) to defined business goals (Table  4 ).

4.2 Step 2: Assessment of indicators and their importance in the automotive industry—presentation of results

According to the step 2 of the research methodology, the assessment of the 20 indicators has been performed by the Authors of this paper. Each of the five overarching assessment criteria (Indicator's features, Relevance for the company, Importance to sustainable business goals, Importance in terms of subsectors, Environmental, Economic, and Social impact) were assessed and presented in a separate table (namely Tables  5 , 6 , 7 , 8 , 9 ).

Based on the above-presented analysis, the last steps of step 2 of the research methodology were applied. The obtained results summarize the assessment in the five overarching criteria:

Criterion 1: Indicator's features—mean value of all assigned points per indicator,

Criterion 2: Relevance for the company—binary value per indicator,

Criterion 3: Importance to sustainable business goals—sum of all assigned points per indicator,

Criterion 4: Importance in terms of subsectors—sum of all assigned points per indicator,

Criterion 5: Environmental, Economic, Social impact—sum of all assigned points per indicator.

They were normalized and the mean value was calculated for each indicator (I.1–I.20). The results can be found in Table  10 .

Based on the above-presented computations, the key indicators for Green PLM were ranked in terms of their importance for the automotive industry from the sustainability-oriented perspective (see Fig.  4 ).

Final ranking of key sustainability-related indicators for Green PLM

Implementation of the two steps of the three-step methodology enabled selection of the key indicators in terms of their importance in the automotive industry and at the same time the indicators that can be used as key sustainability-related indicators for Green PLM development. The five key indicators are:

number of reused parts assembled,

sustainable material,

number of renewable parts,

hazardous material,

non-polluting material.

They have been indicated as the most important and versatile due to the reason that they simultaneously:

match the highest number of sustainability-related business goals,

are relevant for the highest number of environmental, economic, and social impacts,

correspond to the highest number of the automotive subsectors,

achieved the highest scores in terms of desired indicator’s features,

are relevant for the company.

4.3 Step 3: algorithm assessing “greenness” of the product

According to the step 3 of the research methodology, the algorithm assessing “greenness” of the product was applied. It should be noted that the application of the proposed algorithm can differ depending on the need (e.g. promotion/communication aspects), yet it seems that it should be used in the first place as a tool enabling its user to compare a product in its path towards “greenness”. A hypothetical situation can be assumed that a company is developing a product “A” and assigns a number of indicators and corresponding values to assess its “greenness”. Next, after some time, another product is developed—product “B” (e.g. it can be the next generation of product “A”). The company again assigns a number of indicators with corresponding values to assess the level of “greenness” of product “B”. Thus, products “A” and “B” can be compared in terms of their “greenness”. If the level of “greenness” has increased, it can be stated that the company is properly addressing the aim of sustainability and is successfully reaching its business goals. The algorithm presented below has been developed for the five selected indicators in the previous part of the analysis. Nevertheless, it has to be underlined that the algorithm can be easily adjusted for another set of indicators. To demonstrate the application of the proposed algorithm, the following tables have been developed as an example (Tables 11 and 12 ).

According to the above-presented example, once the business goals are listed, the appropriate indicators can be selected (according to the step 2 of the methodology). Next, it can be verified whether an indicator is related to the single part (and thus can be directly assessed whether a part is “green” or not) or is it related to the assembly. So—for example—in indicator I.1 non-polluting material it can be stated that if a part is built of a non-polluting material it can be assessed as 100% green. In case of indicator I.2 hazardous material if material is hazardous it is negative situation. Thus, to assess a part as 100% green it is expected that the value of the indicator I.2 is “no”—meaning that the part is built of non- hazardous material. On the other hand, if we consider the indicator I.8 number of reused parts assembled which does not refer to a part (but to an assembly) the status is n/a. This type of indicator can be considered and analyzed only on the product level (meaning assembly). In order to assess whether a final product is green or not, or how much green it is, certain aggregation and calculations have to be conducted. So initially, following the example of indicator I.1 non-polluting material , all parts in the final product built of non-polluting materials have to be identified. Next, their weights have to be summed up. Subsequently, the total weight of the final product has to be defined. Having the weight of all non-polluting materials in the product and the weight of the final product the proportion of these two can be calculated. In that case, a real percentage value is obtained (e.g. 95.87% for I.1). This value should be compared with the “greenness” threshold which a company should define. If the real percentage value calculated for the given indicator is greater than the “greenness” threshold (so the criterion is met) 1 point can be assigned. For each indicator, a company should set the weights for every criterion (constant values) so that it is clear how important a given indicator for the “greenness” of the whole product is. So for example, the company may assume that indicator I. 2 non-hazardous material (criterion = 1,5) is more important than indicator I.4 sustainable material (criterion = 1,2). Following the algorithm defining the “greenness” of a product, once all the weights of the criteria are set, they can be summed up. Next, for each indicator, a point value resulting from either meeting the “greenness” threshold (1 point) or not (0 points) should be multiplied by the criterion’s weight. The values of these products should be summed. In the final step, the “greenness” status of the product can be calculated by simply dividing these two values. As a result, a percentage value will be defined. Here again, the company should define the critical value—when a product can be considered green (e.g. when the “greenness” status of the product exceeds 75%).

5 Conclusions

Within the literature review three main aspects were analyzed. At first, an analysis focused on the identification of business models corresponding to sustainability was performed. Next, current trends and needs were described in terms of the green PLM. Finally, an introduction describing the automotive industry definition and its subsectors was presented together with defining current challenges. Based on the literature analysis and drawn conclusions, the aim of the research was defined. Thus, the main aim of this paper was to develop a new approach to assessing greenness of a product which would be based on a business goals-oriented algorithm and applied to the new product development phase for the automotive industry. This aim was achieved. Three-step research methodology enabled to assess indicators’ importance for sustainable business goals and business models (step 1), assess indicators and their importance in the automotive industry (step 2) and finally assess “greenness” of the product (3). This complex approach is innovative as it is based on the business-oriented perspective. Selecting suitable business goals and corresponding business models leads to the selection of the appropriate subset of indicators. Several more aspects are taken into account in this holistic analysis, namely indicator's features, relevance for the company, importance to sustainable business goals, importance in terms of subsectors of the automotive industry as well as environmental, economic, and social impact. In the paper the example of analysis was presented. Obtained results led to the selection of five key sustainability-oriented green Product Lifecycle Management indicators the most important in the automotive industry: number of reused parts assembled, sustainable material, number of renewable parts, hazardous material, non-polluting material. According to the third step of the research methodology, the algorithm assessing “greenness” of the product was applied. It enables the company to compute the value of the “greenness” of the product in reference to the key green PLM indicators (selected according to the step 1 and step 2 of the methodology). “Greenness” threshold should be defined by a company according to its internal policy and be in line with the company’s goals and current needs. The exact values of these thresholds may result for example from the expectations of their customers. They can also be imposed by critical values set by the headquarters. Moreover, when setting the “greenness” thresholds, the company should monitor the current corresponding laws and regulations—both national and international. Therefore it is advisable to track regulations based on those mentioned in the Introduction: the European Green Deal, the Fit for 55 package, Circular Economy Action Plan, the Ecodesign Directive, the EU Ecolabel or the EU green public procurement (GPP) and/or join corresponding initiatives, eg.: https://sciencebasedtargets.org/ , https://pacecircular.org/ , https://www.unglobalcompact.org/ ).

5.1 Limitations

The authors would like to acknowledge that the presented analysis has several limitations. The first limitation—the assessment of the indicators was limited to the 20 green PLM indicators described within the paper “Towards GreenPLM—Key Sustainable Indicators Selection and Assessment Method Development” (Helman et al., 2023 ). The points granted in particular parts of the analysis were assigned by the authors of this paper. In the future application of the proposed algorithms, the team assigning the points can be more versatile. The analysis, by definition, was limited to the automotive industry. However, the proposed three-step methodology can be easily adjusted and transferred to another type of industry if needed.

5.2 Further research

Further work will concern issues related to the practical implementation of the methodology developed above. The possibility of building a community of its users in order to maintain and expand the database of green indicators, green products and their components will also be investigated. An interesting issue to research will be the concept of supporting the process of knowledge sharing and cooperation and competition in such a community.

PSS: product-service systems.

Bangsa, A. B., & Schlegelmilch, B. B. (2020). Linking sustainable product attributes and consumer decision-making: Insights from a systematic review. Journal of Cleaner Production, 245 , 118902. https://doi.org/10.1016/j.jclepro.2019.118902

Article   Google Scholar  

Belkadi, F., Bernard, A., & Laroche, F. (2015). knowledge based and plm facilities for sustainability perspective in manufacturing: A global approach. Procedia CIRP . https://doi.org/10.1016/j.procir.2015.01.065

Bocken, N. M. P., Short, S. W., Rana, P., & Evans, S. (2014). A literature and practice review to develop sustainable business model archetypes. Journal of Cleaner Production, 65 , 42–56. https://doi.org/10.1016/j.jclepro.2013.11.039

Boons, F., & Lüdeke-Freund, F. (2013). Business models for sustainable innovation: State-of-the-art and steps towards a research agenda. Journal of Cleaner Production, 45 , 9–19. https://doi.org/10.1016/j.jclepro.2012.07.007

BS 8001:2017 BSI Knowledge. (2017). Retrieved October 4, 2022, from https://knowledge.bsigroup.com/products/framework-for-implementing-the-principles-of-the-circular-economy-in-organizations-guide?version=standard .

Circular Future - Public Affairs and consultancy about Circular Economy. (2022, July 14). Circular Future. Retrieved October 4, 2022, from http://circular-future.eu/

Clinton, L., & Whisnant, R. (2019). Business model innovations for sustainability. In G. G. Lenssen & N. C. Smith (Eds.), Managing sustainable business (pp. 463–503). Springer. https://doi.org/10.1007/978-94-024-1144-7_22

Chapter   Google Scholar  

Commission, E., & Communication, D.-G. for. (2020). Circular economy action plan—For a cleaner and more competitive Europe . Publications Office of the European Union. https://doi.org/10.2779/05068

Council agrees on the Carbon Border Adjustment Mechanism (CBAM). (n.d.). Retrieved October 26, 2023, from https://www.consilium.europa.eu/en/press/press-releases/2022/03/15/carbon-border-adjustment-mechanism-cbam-council-agrees-its-negotiating-mandate/ .

de Medeiros, J. F., Ribeiro, J. L. D., & Cortimiglia, M. N. (2016). Influence of perceived value on purchasing decisions of green products in Brazil. Journal of Cleaner Production, 110 , 158–169. https://doi.org/10.1016/j.jclepro.2015.07.100

Directive 2009/125/EC of the European Parliament and of the Council of 21 October 2009 establishing a framework for the setting of ecodesign requirements for energy-related products (recast) (Text with EEA relevance). , 285 OJ L (2009). Retrieved October 26, 2023, from http://data.europa.eu/eli/dir/2009/125/oj/eng .

Driving Innovation for Automotive Manufacturing Tier 1 Suppliers. (n.d.). Engineering USA . Retrieved September 30, 2023, from https://www.engusa.com/en/posts/driving-innovation-for-automotive-suppliers .

Ellen MacArthur Foundation. (2021). Circularbusinessmodels:redefininggrowthforathrivingfashionindustry. Retrieved October 4, 2023, from https://ellenmacarthurfoundation.org/publications .

Ferasso, M., Beliaeva, T., Kraus, S., Clauss, T., & Ribeiro-Soriano, D. (2020). Circular economy business models: The state of research and avenues ahead. Business Strategy and the Environment, 29 (8), 3006–3024. https://doi.org/10.1002/bse.2554

Fit for 55. (2023, October 10). Retrieved October 26, 2023, from https://www.consilium.europa.eu/en/policies/green-deal/fit-for-55-the-eu-plan-for-a-green-transition/ .

Främling, K., Holmström, J., Loukkola, J., Nyman, J., & Kaustell, A. (2013). Sustainable PLM through intelligent products. Engineering Applications of Artificial Intelligence, 26 (2), 789–799. https://doi.org/10.1016/j.engappai.2012.08.012

Global Industry Classification Standard. (2018). S&P Global Market Intelligence.

Golinska-Dawson, P. (2019). Materials management in remanufacturing of automotive components—A small remanufacturers perspective. IFAC-PapersOnLine, 52 (13), 1738–1743. https://doi.org/10.1016/j.ifacol.2019.11.452

Golinska-Dawson, P., & Pawlewski, P. (2015). Multimodal approach for modelling of the materials flow in remanufacturing process. IFAC-PapersOnLine, 48 (3), 2133–2138. https://doi.org/10.1016/j.ifacol.2015.06.404

Green Public Procurement. (n.d.). Retrieved October 26, 2023, from https://green-business.ec.europa.eu/green-public-procurement_en .

Helman, J., Rosienkiewicz, M., Cholewa, M., Molasy, M., & Oleszek, S. (2023). Towards GreenPLM—Key sustainable indicators selection and assessment method development. Energies, 16 (3), 1137. https://doi.org/10.3390/en16031137

Industry Classification Benchmark. (2022). FTSE Russell. https://research.ftserussell.com/products/downloads/ICB_Rules_new.pdf .

Jonker, J., & O’Riordan, L. (2016). New business models: examining the role of principles relating to transactions and interactions. In H. G. Brauch, Ú. Oswald Spring, J. Grin, & J. Scheffran (Eds.), Handbook on sustainability transition and sustainable peace (Vol. 10, pp. 543–557). Cham: Springer.

Kochańska, J. (2024). Categorization of production losses in the context of assessing production effectiveness indicators. In A. Burduk, A. D. L. Batako, J. Machado, R. Wyczółkowski, E. Dostatni, & I. Rojek (Eds.), Intelligent systems in production engineering and maintenance III (pp. 658–676). Springer. https://doi.org/10.1007/978-3-031-44282-7_50

Kochańska, J., & Burduk, A. (2023). a method of assessing the effectiveness of the use of available resources when implementing production processes. Applied Sciences, 13 (13), 7764. https://doi.org/10.3390/app13137764

Kochańska, J., Musiał, K., Łapczyńska, D., & Burduk, A. (Early Access). The solution of MRSLP with the use of heuristicalgorithms. Bulletin of the Polish Academy of Sciences Technical Sciences; Early Access ; e146407. https://journals.pan.pl/dlibra/publication/146407/edition/128124 .

Kosacka-Olejnik, M., Werner-Lewandowska, K., & Golińska-Dawson, P. (2020). Scanning effectiveness of material flow management in remanufacturing—Case study on diesel particulate filter remanufacturing. Procedia Manufacturing, 51 , 1688–1695. https://doi.org/10.1016/j.promfg.2020.10.235

Łapczyńska, D. (2023). The possibilities of improving the human-machine co-operation in semi-automatic production process. Technologia i Automatyzacja Montażu, 119 (1), 30–36. https://doi.org/10.7862/tiam.2023.1.4

Laubscher, M., & Marinelli, T. (2014). Integration of Circular Economy in Business. https://doi.org/10.13140/2.1.4864.4164

Lewandowski, M. (2016). Designing the business models for circular economy—Towards the conceptual framework. Sustainability, 8 (1), 43. https://doi.org/10.3390/su8010043

Marcon, A., Ribeiro, J. L. D., Dangelico, R. M., de Medeiros, J. F., & Marcon, É. (2022). Exploring green product attributes and their effect on consumer behaviour: A systematic review. Sustainable Production and Consumption, 32 , 76–91. https://doi.org/10.1016/j.spc.2022.04.012

Mentink, B. (2014). Circular business model innovation. MSc thesis , p. 168.

Molasy, M., Rosienkiewicz, M., Helman, J., & Cholewa, M. (2023). Gamification-based crowdsourcing as a tool for new product development in manufacturing companies. In A. Burduk, A. Batako, J. Machado, R. Wyczółkowski, K. Antosz, & A. Gola (Eds.), Advances in production (Vol. 790, pp. 368–379). Springer. https://doi.org/10.1007/978-3-031-45021-1_28

Osterwalder, A., Pigneur, Y., & Clark, T. (2010). Business model generation: A handbook for visionaries, game changers, and challengers . Wiley.

Google Scholar  

Osterwalder, A., Pigneur, Y., & Tucci, C. L. (2005). Clarifying business models: origins, present, and future of the concept. Communications of the Association for Information Systems . https://doi.org/10.17705/1CAIS.01601

Rasmussen, B. (2007). Business models and the theory of the firm. Working Paper , 32: 12

Regulation (EC) No 66/2010 of the European Parliament and of the Council of 25 November 2009 on the EU Ecolabel (Text with EEA relevance). 027 OJ L (2009). Retrieved October 26, 2023 http://data.europa.eu/eli/reg/2010/66/oj/eng .

Ritala, P., Huotari, P., Bocken, N., Albareda, L., & Puumalainen, K. (2018). Sustainable business model adoption among S&P 500 firms: A longitudinal content analysis study. Journal of Cleaner Production, 170 , 216–226. https://doi.org/10.1016/j.jclepro.2017.09.159

Rudnicka, A. (2016). Business models based on sustainability. Practical examples. Modern Management Review . https://doi.org/10.7862/rz.2016.mmr.54

Sanghavi, P., Rana, Y., Shenoy, S., & Yadav, R. (2015). A review on green supply chain management in automobile industry. International Journal of Current Engineering and Technology, 5 , 3697–3702.

Schaltegger, S., Hansen, E. G., & Lüdeke-Freund, F. (2016). Business models for sustainability. Organization & Environment, 29 (I), 3–10. https://doi.org/10.1177/1086026615599806

Sliż, P., & Wycinka, E. (2021). Identification of factors that differentiate motor vehicles that have experienced wear or failure of brake system components during the warranty service period. Eksploatacja i Niezawodność – Maintenance and Reliability, 23 (3), 430–442. https://doi.org/10.17531/ein.2021.3.4

The European Green Deal. (2021). Retrieved October 24, 2023 https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/european-green-deal_en .

TRBC Sector Classification. (n.d.). Retrieved September 30, 2022, from https://www.lseg.com/en/data-analytics/financial-data/indices/trbc-business-classification .

Vila, C., Abellán-Nebot, J. V., Albiñana, J. C., & Hernández, G. (2015). An approach to sustainable product lifecycle management (Green PLM). Procedia Engineering, 132 , 585–592. https://doi.org/10.1016/j.proeng.2015.12.608

Wells, P. (2013). Sustainable business models and the automotive industry: A commentary. IIMB Management Review, 25 (4), 228–239. https://doi.org/10.1016/j.iimb.2013.07.001

Žvirblis, T., Hunicz, J., Matijošius, J., Rimkus, A., Kilikevičius, A., & Gęca, M. (2023). Improving diesel engine reliability using an optimal prognostic model to predict diesel engine emissions and performance using pure diesel and hydrogenated vegetable oil. Eksploatacja i Niezawodność – Maintenance and Reliability . https://doi.org/10.17531/ein/174358

Download references

Acknowledgements

The research leading to these results was carried out within the GreenPLM project funded by transition Technologies PSC S.A.

The research leading to these results received funding from Transition Technologies PSC S.A.

Author information

Authors and affiliations.

Department of Laser Technologies, Automation and Production Management, Wroclaw University of Science and Technology, Lukasiewicza 5, 50-370, Wroclaw, Poland

Maria Rosienkiewicz, Joanna Helman, Mariusz Cholewa & Mateusz Molasy

Instytut Transferu Technologii Sp. z o.o, Na Grobli 15, 50-421, Wrocław, Poland

Transition Technologies PSC S.A, Piotrkowska 276, 90-361, Lodz, Poland

Sylwester Oleszek

Department of Mechanical Engineering, Energetics, Management and Transportation, University of Genoa, Via all’Opera Pia 15/A, 16145, Genoa, Italy

Giovanni Berselli

ADVR, Department of Advanced Robotics, Istituto Italiano di Tecnologia, Via S. Quirico 19d, 16163, Genoa, Italy

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the study in the following way: Conceptualization: M. Rosienkiewicz, M. Cholewa, J. Helman, G. Berselli, Literature review: M. Rosienkiewicz, M. Molasy, J. Helman, Methodology—step 1: M. Rosienkiewicz, J. Helman, M. Cholewa, M. Molasy, Methodology—step 2: M. Rosienkiewicz, J. Helman, M. Cholewa, Methodology—step 3: M. Rosienkiewicz, Formal analysis and investigation: M. Rosienkiewicz, J. Helman, M. Cholewa, Visualization: M. Rosienkiewicz, J. Helman, Writing—original draft preparation: M. Rosienkiewicz, Writing—review and editing: J. Helman, M. Cholewa, M. Molasy, S. Oleszek, G. Berselli, Project proposal development: M. Cholewa,

Scientific advisory: G. Berselli, Funding acquisition: M. Cholewa, S. Oleszek, Project administration: S. Oleszek, Supervision: M. Cholewa, S. Oleszek. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Maria Rosienkiewicz .

Ethics declarations

Conflict of interest.

The authors declare that they have no conflict of interest.

Human and animal rights

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Rosienkiewicz, M., Helman, J., Cholewa, M. et al. Green PLM: business goals-oriented algorithm assessing the greenness of a product in the new product development phase for the automotive industry. Ann Oper Res (2024). https://doi.org/10.1007/s10479-024-06014-4

Download citation

Received : 24 November 2023

Accepted : 16 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1007/s10479-024-06014-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Sustainability
• Product lifecycle management
• Business goals
• Digital manufacturing
• Automotive industry
• Circular economy
• Find a journal
• Publish with us
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 14 May 2024

Carbon carrying capacity in primary forests shows potential for mitigation achieving the European Green Deal 2030 target

• Heather Keith   ORCID: orcid.org/0000-0001-5956-7261 1 ,
• Zoltàn Kun 2 ,
• Sonia Hugh 1 ,
• Miroslav Svoboda 3 ,
• Martin Mikoláš 3 ,
• Dusan Adam 4 ,
• Dmitry Bernatski 5 ,
• Viorel Blujdea 6 , 7 ,
• Friedrich Bohn 8 ,
• Jesús Julio Camarero   ORCID: orcid.org/0000-0003-2436-2922 9 ,
• László Demeter 10   nAff24 ,
• Alfredo Di Filippo 11 ,
• Ioan Dutcă 6 , 12 ,
• Matteo Garbarino   ORCID: orcid.org/0000-0002-9010-1731 13 ,
• Ferenc Horváth 10 ,
• Valery Ivkovich 14 ,
• Āris Jansons 15 ,
• Laura Ķēņina 15 ,
• Kamil Kral   ORCID: orcid.org/0000-0002-3848-2119 4 ,
• Dario Martin-Benito   ORCID: orcid.org/0000-0002-6738-3312 16 ,
• Juan Alberto Molina-Valero   ORCID: orcid.org/0000-0002-8359-5761 3 ,
• Renzo Motta   ORCID: orcid.org/0000-0002-1631-3840 13 ,
• Thomas A. Nagel 3 , 17 ,
• Momchil Panayotov 18 ,
• César Pérez-Cruzado 19 ,
• Gianluca Piovesan   ORCID: orcid.org/0000-0002-3214-0839 20 ,
• Cătălin-Constantin Roibu 21 ,
• Pavel Šamonil 4 ,
• Ondřej Vostarek   ORCID: orcid.org/0000-0002-0657-0114 3 ,
• Maxim Yermokhin   ORCID: orcid.org/0000-0002-7317-3995 22 ,
• Tzvetan Zlatanov 23 &
• Brendan Mackey   ORCID: orcid.org/0000-0003-1996-4064 1  

Communications Earth & Environment volume  5 , Article number:  256 ( 2024 ) Cite this article

2 Altmetric

Metrics details

• Carbon cycle
• Climate-change ecology
• Climate-change mitigation
• Forest ecology

Carbon accounting in the land sector requires a reference level from which to calculate past losses of carbon and potential for gains using a stock-based target. Carbon carrying capacity represented by the carbon stock in primary forests is an ecologically-based reference level that allows estimation of the mitigation potential derived from protecting and restoring forests to increase their carbon stocks. Here we measured and collated tree inventory data at primary forest sites including from research studies, literature and forest inventories (7982 sites, 288,262 trees, 27 countries) across boreal, temperate, and subtropical Global Ecological Zones within Europe. We calculated total biomass carbon stock per hectare (above- and below-ground, dead biomass) and found it was 1.6 times larger on average than modelled global maps for primary forests and 2.3 times for all forests. Large trees (diameter greater than 60 cm) accounted for 50% of biomass and are important carbon reservoirs. Carbon stock foregone by harvesting of 12–52% demonstrated the mitigation potential. Estimated carbon gain by protecting, restoring and ongoing growth of existing forests equated to 309 megatons carbon dioxide equivalents per year, additional to, and higher than, the current forest sink, and comparable to the Green Deal 2030 target for carbon dioxide removals.

Similar content being viewed by others

Integrated global assessment of the natural forest carbon potential

Russian forest sequesters substantially more carbon than previously reported

Mature Andean forests as globally important carbon sinks and future carbon refuges

Introduction.

International negotiations over climate change have reached agreements and targets - Paris Agreement 1 and Glasgow Climate Pact 2 - with the goal of stabilising atmospheric CO 2 concentration and temperature increase to well below 2 °C above pre-industrial levels and pursue efforts to limit warming to 1.5 °C 3 . However, operationalising these commitments to achieve a true reduction in the carbon stock in the atmosphere requires re-thinking some fundamental concepts of carbon accounting, revising specific guidelines and rules, and collecting the necessary data. The land sector is particularly problematic where flows of carbon between the biosphere and atmosphere are naturally two-way and the reservoirs that store carbon in the biosphere differ in their ecosystem condition, in terms of stability, longevity, and resilience to disturbances. Current accounting methods do not distinguish these differences in quality and the gross flows of emissions and removals adequately in reporting of net emissions reductions towards targets. Unintended consequences of mitigation actions have resulted, such as emissions from forest harvesting being netted out against removals from the entire forest area, carbon stocks in forests having different levels of ecosystem condition not being differentiated, the historical debt in carbon stock due to human activities is a permanent loss not being counted, and the effect of the time difference between instantaneous emissions and future removals by tree growth on the resultant carbon stock change in the atmosphere not being included in life-cycle assessments 4 , 5 , 6 , 7 , 8 .

We address a key issue in accounting concerning the reference level from which past changes in forest carbon stocks are calculated and future changes are predicted. Using an “ecologically-based reference level” for forest ecosystems is crucial for ensuring consistent information regarding: (i) assessing the carbon stock loss that has occurred in the past due to human activities; (ii) predicting the potential gains in stocks by changing forest management; and (iii) determining the foregone mitigation benefits due to managing forests at carbon stocks below their maximum. This context using a carbon stock-based target provides an alternative accounting solution which can be implemented under the Paris Agreement.

Management of forests is critical for mitigation in terms of both avoiding emissions to, and increasing removals from, the atmosphere. Forest ecosystem are large stores of carbon (~660 PgC in living and dead biomass and soil organic carbon 9 and have high potential to sequester additional carbon (with forests constituting most of the terrestrial sink 10 ). Systems for evaluating the mitigation outcomes of forest management strategies, including all carbon stocks and flows through forests, products and energy, are thus crucial and require consistency in monitoring and accounting for changes in stocks. A consistent reference level has been an issue particularly prevalent in continents with a long history of land use, such as Europe, where the effect of human activities on ecosystem carbon stocks is difficult to define because of the transformations of primary forest into managed or degraded ecosystems 11 , 12 . Describing the reference level in terms of an ecological state across all forest areas and management systems provides consistency for interpretation of carbon stock changes in the long-term, rather than an historical, business-as-usual or near-term projected level. Reporting in greenhouse gas inventories is only required for managed lands 13 , although unmanaged forests can be included, and for the European Union, 98% of forest area is monitored and reported 14 (S1.1).

The current carbon stock (CCS) of forest ecosystems is a function of natural history, environmental conditions, and natural disturbance regimes, but also human land use, particularly the harvesting and regeneration of trees. The reference level used currently under the UNFCCC to assess each country’s compliance against emissions reduction targets (previously under Kyoto Protocol rules or European LULUCF Regulation), projects future net annual emissions based on the CCS together with forecasted dynamics resulting from management as either the continuation of documented historical practices or future implementation of approved policies 15 . Using this approach to the reference level may show changes in net emissions due to human activities. However, it cannot be used to predict the potential carbon stock that forests could store if management changed to allow restoration of the maximum stocks for the site conditions given the natural potential of forest ecosystems. To answer the question of potential stock requires an ecologically-based local reference level derived from a forest ecosystem’s carbon-carrying capacity (CCC).

We show that estimating the carbon stored in a primary forest represents a given forest ecosystem type’s CCC - defined as the mass of carbon stored in an ecosystem at landscape scales given the life history traits of the tree species (e.g. longevity), prevailing environmental conditions (noting these are varying due to climate change), and the impacts of natural disturbance regimes, but excluding direct anthropogenic disturbance 16 . The stock at CCC is the potential stock that can be maintained by natural processes within an ecosystem in a resilient and self-sustaining manner. The residence time of the stored carbon is centennial scales, with cycling of carbon maintained by frequent low- to moderate-severity natural disturbances, as well as some infrequent larger-scale disturbances, such that the carbon stocks remain relatively stable when averaged over long spatial and temporal scales inclusive of the regenerative capacity of ecosystems 12 , 17 . However, carbon stocks and relative proportions among ecosystem compartments may be changing with changing climate.

Primary forests are defined by the FAO 18 as naturally regenerated forest of native tree species, with no clearly visible indications of direct human activities, and whose composition, structure and dynamics are dominated by ecological and evolutionary processes, including natural disturbance regimes. Resulting forests include a range of tree ages and seral stages at stand and landscape scales (S1.2). In Europe, the long history of land use means that the term is more appropriately interpreted as referring to forests that are long unlogged and have reached a level of maturity including many with old-growth characteristics and a high degree of naturalness and ecological functioning, without implying that there was never human disturbance 19 , 20 , 21 . Compared to secondary forests managed for commodity production, these primary forests have the highest levels of ecosystem integrity 22 and store the largest carbon stocks for a given forest type and environmental conditions, because of the relationship between high biomass in large trees and total biomass density 23 , 24 , 25 , 26 .

We present the theoretical basis and data to support the proposition that using the CCC in primary forest as the ecologically-based reference level can improve accounting for past and predicting future changes in forest carbon stocks. Changes can then be tracked in the condition of forests resulting from increases/decreases in their carbon stocks due to human activities and management strategies. This accounting would improve the transparency and soundness of the predicted mitigation potential of forests because the reference level is defined as an ecological state rather than a construct of human activities that can vary depending on timeframe, location and management legacy. We address this proposition through two research questions.

What is the CCC of forest types in Europe that is represented by the extant primary forests and how well are these carbon stocks represented by global default values and models?

What is the potential contribution to climate change mitigation by protecting primary forests to avoid carbon stock losses, and restoring secondary forests to gain carbon stocks?

The foci of our study are the primary forests of Europe which serve as a case study of how our approach could have global applications. Extensive human land use (S1.1) present challenges to identifying primary forests. Even though these identified forest areas have least disturbance and highest levels of ecosystem integrity, our estimates of CCC are uncertain and may be conservative. The concept of CCC has not been applied in Europe nor has it been used as the reference level for carbon accounting. The data on primary forests presented here now make this approach to accounting feasible. We found that the large carbon stocks in primary forests have been underestimated previously. Applying an ecologically-based reference level of the CCC revealed the foregone losses due to maintaining forests at carbon stocks below their maximum, and the mitigation potential from gains by ongoing forest growth. Hence, the protection and restoration of primary forests is a critical action for climate mitigation.

Carbon stock density in primary forests across Europe

The extant forest area of Europe was studied with field sites located in areas of primary forest from which carbon stocks were calculated (Fig.  1 ). Above- and below-ground biomass and dead biomass for primary forest sites were aggregated by country and forest type (Fig.  2 ). To complete the carbon budget, soil carbon data for the site locations was extracted from the global map because there was insufficient soil data from the field measurements (Supplementary Fig.  S5 ). Biomass carbon stock varied by an order of magnitude across countries, from the lowest in alpine birch forest in Sweden (21 MgC ha −1 ) to the highest in mixed spruce-fir-beech forest in Bosnia-Herzegovina (346 MgC ha −1 ) (Fig.  2 ). Variation in dead biomass was not coincident with living biomass; it is influenced by the total amount of biomass but also disturbance factors causing mortality. For example, the highest amount of 113 MgC ha −1 in Czech Republic conifer forest was measured after a severe windstorm (Fig.  2 ).

Primary forest sites included (i) research sites and (ii) sites reported in the literature that were selected by observation of ecological characteristics, and (iii) forest inventory sites described as primary forest or as natural forest >100 years old and under some protection status. Forest cover was defined by area of CORINE forest types 94 and forest canopy cover ≥30% (S1.7) (See Methods: Identification of primary forest sites).

Coloured bars are the mean value and lines are standard deviations for the total biomass. Countries ordered approximately by latitude. Numbers for each column refer to the numbered rows in Supplementary Table  S2 . Forest types classified by the European CORINE Land Cover 94 and then by dominant species according to national systems. Data from national forest inventories (NFIs) were used for Sweden (1, 2, 3), Germany (8), and Romania (25) (no dead biomass data) and so do not have the same level of strict definition of primary forest (see S1.2).

Classified by Global Ecological Zones (GEZs) 27 (Supplementary Fig.  S1 ), biomass carbon stocks were similar in temperate and subtropical (Mediterranean) zones, and lower in the boreal. Mountain systems had higher biomass than continental forests. Increasing biomass occurred from conifer to broadleaf and mixed forest types within the temperate and subtropical zones, but the reverse in the boreal (Fig.  3 ).

Within GEZs, forest types are classified by the European CORINE Land Cover 94 (conifer, broadleaf, mixed forest). Coloured bars are mean value from the primary forest site data, lines are standard deviations, and the numbers in each column are the number of sites per class.

Distribution of sites within the environmental space for forest cover within each GEZ illustrates their representativeness in terms of productive potential (Fig.  4 ). The sites are generally at higher elevation (except inventory sites in temperate zones and all sites in temperate continental and oceanic), sites have lower temperatures in the boreal and subtropical mountain zones, and research sites are drier and hotter in temperate oceanic forest.

Environmental space defined by elevation, mean annual temperature and water availability index (S1.5) for each GEZ. The environmental space is shown by 2000 random points within the current forest cover that occurs within each GEZ. Primary forest sites distinguished as research sites, literature sites and forest inventory sites.

Comparison with global modelled and default biomass

The spatial distribution of global modelled aboveground biomass (Supplementary Fig.  S2 ) and soil carbon (Supplementary Fig.  S3 ) show high carbon stock densities in some of the forest areas mapped as remaining primary forest 28 (Supplementary Fig.  S4 ), such as Carpathian Mountains in eastern Europe, Dinaric Mountains in the Balkans, and Pyrenees in north-eastern Spain. High soil carbon contents are related more to cooler temperatures and high elevation, particularly where peat or transitions occur.

Frequency distributions of biomass carbon stock density at the sites were consistently higher than the modelled data from the global biomass maps across all forest types, with the greatest difference for mixed species, then broadleaf, and least for conifers (Fig.  5 ).

Distribution of carbon densities compared for the site data and global modelled data from GlobBiomass 97 and GeoCarbon 98 for the primary forest site locations, for a all forest types, b conifer species, c broadleaf species, d mixed species. The dataset for all forest types included sites of unidentified forest types and thus was larger than the combined broadleaf plus conifer plus mixed datasets. Results for paired two sample t -test one-tail for the site level data are presented for each forest category to show the statistical differences. All forests: Site data and GlobBiomass t  = 59.47, P  < 0.0001, n  = 5942. All forests: Site data and GeoCarbon t  = 53.16, P  < 0.0001, n  = 3778. Conifer: Site data and GlobBiomass t  = 20.70, P  < 0.0001, n  = 720. Conifer: Site data and GeoCarbon t  = 19.80, P  < 0.0001, n  = 597. Broadleaf: Site data and GlobBiomass t  = 29.55, P  < 0.0001, n  = 1176. Broadleaf: Site data and GeoCarbon t  = 38.21, P  < 0.0001, n  = 774. Mixed forest: Site data and GlobBiomass t  = 24.24, P  < 0.0001, n  = 182. Mixed forest: Site data and GeoCarbon t  = 33.61, P  < 0.0001, n  = 178.

The cumulative biomass from the global models was lower than the site data by proportions of 0.57 for GeoCarbon and 0.59 for GlobBiomass for all forest types, 0.64 and 0.65 for conifer species, 0.53 and 0.61 for broadleaf species, and 0.28 and 0.46 for mixed species, respectively (Supplementary Table  S5 ). The cumulative biomass from the different data sources showed the greatest difference for research and literature sites, and least for inventory sites (Supplementary Fig.  S6 ). The difference was least in Sweden and greatest in the carbon-dense forests, such as Montenegro, Bosnia-Herzegovina, Romania, Albania and Bulgaria (Supplementary Fig.  S7 ).

Biomass carbon stocks at primary forest sites classified by GEZ compared with global modelled biomass (GlobBiomass) within the mapped primary forest area of the GEZ 28 were similar for boreal forests, 2-times higher for temperate, and 2.5–3-times higher for subtropical forests (Fig.  6 , Table  1 ). Comparison for each site location showed the field data were 3-times higher on average than the global modelled biomass.

Coloured bars are mean value from the primary forest site data within each GEZ and lines are standard errors. Primary forest sites distinguished as research sites, literature sites and forest inventory sites. Default values include the IPCC within GEZs 29 (primary forest values for Europe were used for the temperate GEZs, no data were available for boreal mountain, data from the Americas was used for boreal coniferous, data for Asia for subtropical mountain, and average data for all age categories from Asian, Americas and African for subtropical dry), the modelled GlobBiomass within GEZs 97  and the mapped area of primary forest 28 , and the Forests Europe average for all forest types 81 , 99 .

Field measurements of biomass at sites showed higher carbon stocks than IPCC default values within GEZs 29 (Fig.  6 ), with more than twice the stock in temperate oceanic forest, but similar or slightly lower in temperate continental forest. The database presented here consists of many more site measurements than were used in the IPCC guidelines, which have uncertainty levels of 90% and many data gaps (references cited for GEZs in Europe include eight research studies, one inventory study and three remote sensing studies, with only two studies in primary forests). The 2019 updated default values are an improvement to the 61 MgC ha −1 used in the original 2006 guidelines 30 and now recognise that carbon stocks are higher and the importance of differentiating GEZs and primary and secondary forests.

Tree size distribution

Distributions of biomass carbon stock density and tree density in relation to tree diameter size class show the contributions of each size class to the total carbon stock of the forest (Fig.  7 , Supplementary Fig. S8 , Supplementary Table  S6 ). The highest density of tree numbers occurred in the smaller diameter classes with a long positive skew. In contrast, carbon stock density had a normal to positive skew distribution, with a small number of large trees having high stocks. Cumulative biomass showed the proportion of the tree size range that contributed to the total biomass carbon stock. Across all primary forest sites sampled, 50% of total carbon stock in living biomass was contained in trees with DBH ≥ 60 cm (median value), but this represented only 15% of the number of trees. (Tree density depended on the minimum diameter measured, commonly 5 cm, but varied among countries from 1 to 10 cm) (Supplementary Table  S2 ). The proportion of total biomass in the primary forest contributed by large trees varied depending on forest type (broadleaf, conifer and mixed forest), and DBH threshold, with ranges in biomass carbon stocks from 23 to 43% at the high tree DBH threshold to 68–75% at the low threshold (Supplementary Figs.  S8 and S9 ).

The light green area represents the distribution of tree density, the dark green histogram represents the distribution of total live biomass carbon stock, the red curve represents the cumulative biomass over the range in tree sizes, the blue dashed line represents the proportion of total biomass carbon stock contributed by large trees (DBH ≥ 60 cm). Vertical bars are standard errors.

Total carbon stocks and potential stock gain

The current total forest area in Europe is 174 Mha and 1.07% is classified and mapped as primary forest 28 (Supplementary Fig.  S4 ), with 78% in the boreal region. The aboveground biomass carbon stock in the total forest area is estimated to be 9861 TgC (based on current data and assumptions), and the contribution of primary forests is 1.02%. The total carbon stock in the area mapped as primary forest 28 was estimated from the collated primary forest site data and this represents the CCC (see Methods: Spatial estimation of carbon stocks and stock changes and Supplementary Information  S1.8 ). This updated carbon stock was 1.6-times more than that currently estimated from global modelled biomass within the mapped primary forest area, with temperate and subtropical forests double or more, and boreal forests similar stocks (Table  2(1) , Supplementary Table  S9 , Fig.  6 ). However, the boreal and subtropical dry (Mediterranean) biomes have low numbers of sites.

The potential gain in carbon stock in secondary forests by ongoing growth was calculated as the increasing carbon stock over time from CCS towards its CCC. The estimated CCC in these secondary forest areas was 2.3-times the current stock with an additional 12,659 MtC, indicating the potential for high carbon sequestration through restoration (Table  2(2) , Supplementary Table S9 ).

The area of mapped primary forest 28 is likely on overestimate in some countries, such as Sweden, Finland and the Iberian Peninsula, based on observations of researchers (S1.7). Such overestimates in area suggest that the high carbon stock densities from the site data assigned to areas of primary forest would result in a higher total carbon stock for primary forest. Conversely, the effect of mapping some primary forest that should have been assigned to secondary forest, is a larger area for potential gain in carbon stock. This would amplify the result of mitigation potential in Table  2(2) . Additionally, if the area of primary forest is even smaller than that mapped, this amplifies the significance of their rarity, and hence, importance of their protection.

Carbon stocks in primary forests have been underestimated

The larger carbon stocks identified at primary forest sites compared to estimates based on modelled or default values is consistent with that found in individual forest types in Europe 31 . Our updated estimate of CCC means the mitigation benefit of primary forests by protecting their accumulated carbon stocks and avoiding their loss has been undervalued in climate and forest management policies.

The main reason for the discrepancy in estimates of aboveground biomass is that the global models based on remotely sensed data have high uncertainties at high biomass densities (>125 MgC ha −1 ) where the retrieval was effectively based on a single radar observation 32 . Comparison of the biomass map with independent reference data concluded from the validation process that biomass was underestimated at ≥75 MgC ha −1 in the temperate forest biome 33 . The models use plot data to calibrate the satellite-based estimates of extant forest cover and thus estimate CCS. However, our results from the site data show that even in the locations where primary forests exist and their structural characteristics should be detected by remote sensing, those calibrations do not accurately reflect observed carbon stocks (Fig.  5 ). The modelled data lacks differentiation between forest types (conifer, broadleaf, mixed) suggesting that the models are not able to account adequately for the different biomass densities of these forest types. Consequently, underestimation of biomass occurs particularly for broadleaf species.

Biomass estimates using plot data also have limitations, particularly in fragmented landscapes. First, the area sampled is a very small proportion of the area of primary forest and total forest cover. Second, the representativeness of plot data for ecosystem types at the landscape scale is uncertain as such small proportions of natural forest ecosystems remain. Third, the distribution of sites is spatially biased, and some forest types are poorly represented, such as subtropical dry and mountain systems, coastal boreal and coastal temperate oceanic systems. Therefore, the full distribution of forest types and their carbon stocks are not evident. Fourth, patch heterogeneity is a characteristic of old-growth forest structures but measuring the full extent of spatial structural complexity of forests, including canopy gaps and variations in stand density related to fine-scale natural disturbance events, is challenging with plot-based sampling, as seen by the high variance around mean plot values for a forest type. However, carefully randomised and replicated plot locations have been shown to record robust estimates of biomass 34 , 35 , 36 .

Three additional factors are likely to amplify the underestimation of biomass from global models. First, carbon stock densities estimated from site data may be lower than the forest type within GEZ average because the patches of primary forest remaining often occur lower on the productivity gradient because they are less accessible or desirable for human intervention 37 (Fig.  4 ). However, there may be cases of sites preferentially sampled to select large growth forms and areas of productive forests selectively preserved for private use historically 38 , 39 . Particularly in Europe with the long history of land use (S1.1), the available forest areas with least disturbance likely do not fully represent the CCC of the forest type, which means estimates of the potential carbon storage may be conservative. Second, total carbon stock includes not only aboveground living biomass but also belowground biomass and dead biomass of standing dead trees, coarse woody debris, litter and soil. These components are higher in primary forests with old trees and complex structures 40 . The importance of including dead biomass and belowground biomass is demonstrated, adding on average 16% and 22% to the aboveground living biomass, respectively (averages from Fig.  2 ). Improved data are needed for soil carbon. Third, most plot-based forest biomass data are derived from allometric equations. Sampling of trees to derive these equations often underrepresents large trees because destructive measurement of total biomass is difficult and often done in production forests that do not contain large trees. Underestimation has been demonstrated in the biomass of forest plots derived from allometric equations compared with detailed three-dimensional laser measurements across the full range of tree sizes 41 , and this increased with tree size 42 . Extrapolation of biomass calculated for trees larger than the calibration data increases uncertainty in biomass estimates, thus particularly affecting results for primary forests.

The large carbon stocks in primary forests are due to the presence of large old trees, where tree size is relative depending on forest type and environmental conditions. These large trees are rarely present in forests managed for commercial production by clear cutting, and reduced under selective cutting, as the trees only attain harvest maturity and not the large sizes of old trees, as well as their associated dead biomass 43 , 44 . Within the production cycle, the remaining old forests (for example, >100 years old with a high density of moderately large trees) may attain similar biomass density to that of long-unmanaged forests 31 , but this represents a small proportion of the total production forest area. Our results provide a comprehensive assessment of the contribution to carbon stocks by large tree-size classes for primary forests in all biomes across Europe 24 , 45 , 46 , 47 , (Supplementary Table  S8 ).

While modelled and default values of carbon stocks have been refined over the last decade, our synthesis of site data for primary forests within GEZs across Europe (Table  1 , Supplementary Table S9 ) further improves the input data for deriving biomass carbon stocks and contributes to the next update of IPCC default values.

Mitigation potential from protecting and restoring primary forests

Protecting and restoring primary forests contribute to climate mitigation by (i) retaining an accumulated stock of carbon in living and dead biomass and soil, (ii) maintaining the natural terrestrial carbon sink to buffer some of the impact of elevated atmospheric CO 2 concentration, and (iii) removing CO 2 from the atmosphere through restoration of secondary and degraded forests 43 .

(i) The priority for mitigation action is retaining carbon stored in forests and avoiding emissions. Thus, the mitigation benefit of primary forests is the magnitude and longevity of their accumulated ecosystem carbon stocks. The large amounts of carbon stored in primary forests, as revealed in this study, represent long-lived, stable and resilient stocks in ecosystems with a high level of integrity 22 . Primary forest ecosystems have lower risk of emissions to the atmosphere caused by either human activities or increases in extent, frequency, and severity of disturbance events 48 , compared with forests that have lower levels of ecosystem integrity, such as secondary forests or plantations. Area and biomass harvested annually, and use of wood resources, have been increasing since 2016, thus reducing carbon stocks, although rates vary between different data sources 49 , 50 , 51 , 52 , 53 . A documented increase in canopy mortality of 2.4% over the last three decades was attributed to intensified tree harvesting as the prime agent and correlated with reported wood removal statistics 54 . Disturbance regimes under the influence of climate change have been intensifying, for example windstorms, fires, insect outbreaks, and are likely to impact carbon storage 55 , although much of the effect is transference from living to dead biomass 56 . The biodiversity, structural complexity and ecological and evolutionary processes operating in primary forest ecosystems confer greater stability, resilience and adaptive capacity that support accumulation and maintenance of carbon stocks 57 , 58 , 59 , 60 . Hence, primary forests provide the benefit of avoided damages.

(ii) Global forests act as sinks with positive net ecosystem productivity as trees continue to grow, accumulate biomass, produce dead biomass, and store soil carbon 61 , 62 , 63 . In general, secondary forests have higher rates of biomass productivity because their management aims to increase wood production by maintaining trees at younger ages, often increasing plant-available resources, sometimes selecting more efficient genetics, and locating on more productive sites 61 , 64 . However, a continuing sink has been demonstrated in older forests through many studies, both at the stand scale and from tree ring analysis 12 , 40 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , and rates of net primary productivity can be similar in primary and production secondary forests 63 . The complex heterogeneity of older forests maintained by natural disturbance regimes contribute to their high rates of productivity by creating structural diversity of tree sizes, ages, spatial distribution, vertical canopy structure, leaf area index and accumulation in the soil carbon pool 72 , 73 . The carbon sink in forests globally absorbs approximately 18% of annual anthropogenic CO 2 emissions (2001–2019) 74 and unharvested mature forests account for about half of this sink 75 .

(iii) Ecological restoration of secondary and degraded forests provides a pathway for removals of atmospheric carbon, through gains in carbon stocks with growth toward the CCC and reconnecting fragmented landscapes 4 , 59 , 76 , 77 . The ecologically-based reference level of CCC that is characterised by primary forests provides information about the composition, structure, ecological processes and natural disturbance regimes of these ecosystems that can help guide restoration activities and the selection of appropriate indicators for assessing progress.

The potential carbon stock gain by allowing trees to continue growing beyond currently applied harvest maturity was estimated from the analysis of tree size distribution (Supplementary Fig.  S8 ). The carbon stock foregone by harvesting at a given tree size was simulated by restricting trees at or below the harvest diameter threshold but maintaining the same tree density. The foregone stock varied from 12 to 21% at the high tree diameter to 46-52% at the low tree diameter (Supplementary Table  S7 ). The biomass carbon stock of forests managed on a harvesting rotation would be lower than the estimate here when the full age distribution of stands since regeneration are included at the landscape scale. Hence, this estimate of foregone carbon stock is conservative.

This potential for carbon stock gain has been demonstrated in other studies with forests naturally developing following protection from human intervention 78 , 79 . Historical change in forests over the last 250 years in Europe shows 41.7 Mha being converted to production management involving wood extraction and tree species conversion. This has resulted in a 38% average reduction in biomass carbon storage 80 , observed under a range of conditions 63 , 81 , 82 , (Supplementary Table  S9 ). The ensuing wood products constitute a small pool of 10% or less of the carbon stored in aboveground living forest biomass 83 , 84 , 85 , and have a shorter lifetime than that of trees 30 . However, increasing the proportion of long-lived wood products and their longevity, reducing waste on-site and during processing, and substituting wood for products with large carbon footprints due to their processing such as concrete, would contribute to increasing carbon storage compared to current forest management.

We estimated the potential for carbon stock gain by regrowing secondary forests to be more than double the current stock, with an additional 12,659 MtC (46,415 MtCO 2 ) (Table  2(2) ). Similar results have been reported, including modelling of European forests using estimated maximum carbon stock in broadleaf forests of 170 tC ha −1 (compared to our site measurements of 187 tC ha −1 ) and in conifer forests of 130 tC ha −1 (current study 145 tC ha −1 ) 86 , and estimated 42–47% carbon stock loss due to degradation in the land sector 25 . Based on historical and ecological sources, carbon storage of temperate deciduous/ mixed forests was estimated at about half of the potential storage 87 . Further work is required to refine these estimates, for example using site matching between secondary and primary forests, improved representativeness of sites within forest types, and finer spatial scale of aggregation for the comparison. The actual gains in carbon stock in the future will, however, depend on the impacts of climate change on forest productivity and the relative rates of carbon fluxes through the vegetation and soil that determine carbon accumulation, its stability and resilience.

The potential mitigation benefits from protecting and restoring forests can be considered relative to the overall European target for net annual removals in the LULUCF sector under the Fit for 55 package in the European Green Deal of 310 MtCO 2_e by 2030 14 and the simulated trajectory for forest land of 410 MtCO 2_e removals required to reach this target 88 . Our estimated gain in carbon stock by regrowing secondary forests (Table  2(2) ) equates to an annual rate of removals of 309 MtCO 2_e ; assuming the CCC would be achieved after 150 years with a linear rate of accumulation (or 464 MtCO 2_e after 100 years). (Carbon accumulation is likely to have a logarithmic form, in which case annual removal rates would be higher initially and then slow but reaching the same stock). These estimated removals are additional to, and higher than, the current forest sink in the EU (2021: 289 MtCO 2_e 89 . Thus, restoration of the existing forest area through carbon accumulation by ongoing growth could achieve the net removals target in the land sector and fill the carbon debt created by historical forest management. However, all emissions reduction activities should be pursued and the requirement to maintain wood provisioning is recognised. Our results demonstrate the considerable opportunities for increasing carbon storage in the existing forest area, for example through restoration of 30% of forest area (EU Biodiversity Strategy target of 30% protection 90 , protecting biodiversity habitats, optimising wood production, retaining large trees, restoring coppices, and lengthening rotation periods, as well as opportunities for increasing the forest area through reforestation and regeneration (cognisant of requirements for other land uses, natural ecosystems and ecosystem services).

Conclusions

Retaining and increasing carbon stocks in forests and avoiding emissions should be recognised as priorities for global environmental policy progressing towards more ambitious climate and biodiversity targets for the European Green Deal. Achieving these targets requires conservation management and governance to ensure formal and enforced protection that prevents degradation by logging and mining, encroachment by roads and other infrastructure developments. In Europe, only just over half of the identified primary forests are strictly protected 81 , 91 and degradation continues in some places even within designated protected areas 37 . Under the EU Biodiversity Strategy for 2030, the aim is to strictly protect all remaining primary and old-growth forests, restore and manage them and their buffer zones for biodiversity conservation 92 . This protection is a mitigation activity that can be implemented rapidly and contribute to many co-benefits for biodiversity conservation and provision of ecosystem services, cognisant of their complementarities and interdependencies 17 , 93 . Europe has few areas of forest with minimal human disturbance and so protecting these remaining areas should be a priority for climate and forest policy. Restoration of forest ecosystems by allowing continued growth of regenerating forests, active restoration measures, and re-connecting fragmented remnants across landscapes, will provide crucial mitigation benefits that contribute to emissions reduction targets as well as existing and future co-benefits.

Identification of primary forest sites

The status of primary forests in Europe is described (S1.1) and the characteristics used for their identification detailed (S1.2, Supplementary Table  S1 ). As much data as possible was collated from existing studies in primary forest areas in boreal, temperate and subtropical (Mediterranean) zones within Europe. Site data were collected from three sources: (i) research sites were from existing studies (the network used for mapping primary forest in Europe by Sabatini et al. 28 ), and the primary forest status was determined from observed characteristics (Supplementary Table  S2 ); (ii) site data from the literature where characteristics of the forest are described (S1.4, Supplementary Data  1 ); and (iii) inventory data were obtained from NFIs where the data custodians filtered the database to extract sites that were described as primary forest, or as natural forest under some protection status, and stands older than 100 years (Supplementary Table  S2 , Fig.  1 ). The inventory sites do not necessarily comprise all the characteristics of primary forests but provide an estimate of maximum carbon stocks. Site selection was based on observation of forest characteristics with the assumption that there was no systematic selection for site conditions. However, this synthesis of sites is currently the most comprehensive compilation of primary forest site data. The database of primary forest sites collated consists of 1818 research sites with 242,461 trees across 16 countries; 6015 inventory sites with 45,801 trees across 3 countries; and 149 literature sites across 17 countries.

The locations for these sites are not necessarily representative of all remaining primary forests or forest types and the distribution of sites across Europe shows a spatially biased sample (Fig.  1 ). Patches of primary forest remaining are often located on extreme sites, for example steep slopes, shallow soils and high elevation, that are likely to be at the lower end of the productivity gradient for the forest type. To test the representativeness of the sites, they were located within the environmental space for current forest cover within each GEZ defined by elevation, mean annual temperature and a water availability index (S1.5).

Sites were classified by forest type - broadleaf, conifer, mixed forest – (according to the European classification from Corine Land Cover 2018 94 ), and Global Ecological Zones (GEZ 27 ) (Supplementary Fig.  S1 ) for the purpose of aggregation and reporting of results and for comparability with that used by IPCC.

Carbon stock components

The basic inventory measurements at all sites included the DBH of all trees within a defined plot area, tree height (in some cases), species or forest type, living and dead trees, and dimensions of coarse woody debris. Individual tree biomass was calculated for above- and below-ground biomass of living and dead standing trees, and biomass of coarse woody debris (or lying dead wood).

Tree dimensions were converted to biomass using regional and species-specific allometric equations where available. Wood volume of stems and coarse woody debris was converted to biomass using species-specific wood density and carbon concentration, and wood decay classes where available. Methods used for each dataset are described in Supplementary Table  S1 . Additional data for dead wood biomass in various forest types in Europe was sourced from the literature (Supplementary Table  S3 ). Data for soil organic carbon was sourced from global modelled data (GSOC 95 , S1.7).

The relationship between tree size and biomass carbon stock for individual trees was analysed where individual tree data were available for the sites. The distribution of biomass carbon stock in the primary forests was differentiated according to the stock contributed by trees larger than the harvesting diameter threshold, that is the large trees that would not occur in a forest managed on a harvesting rotation. The size of trees at harvest varies depending on many factors, including species, environmental growth conditions, silvicultural systems, industry regulations or protocols, products and their markets. Harvesting may be determined by age or size of the trees and ranges are reported (Supplementary Table  S4 ). As a conservative estimate of the tree size at harvest, we selected a minimum and maximum for each forest category: 50–80 cm for broadleaf, 40–60 cm for conifers, and 50–70 cm for mixed forest. These tree sizes relate to single-stemmed trees or high forest and the harvesting of coppice forests at smaller sizes has not been analysed here, although a common silvicultural practice in Mediterranean forests 96 .

The biomass contributed by the large trees is reported as a percentage of the total biomass in the primary forest. The carbon stock that is foregone by harvesting was estimated as the difference in biomass between the primary forest and the same number of trees but limited to the diameter threshold at harvest. In the harvesting scenario, the biomass of all trees per hectare were included but the maximum size was set at the threshold DBH representing harvest maturity. Calculated biomass included all trees below the diameter threshold plus the biomass of the equivalent number of trees above the threshold assuming their size was at the threshold, that is, at harvest maturity or rotation forest age. The foregone stock was represented as the percent of the biomass carbon stock for a range of tree sizes at harvest maturity.

Spatial estimation of carbon stocks and stock change

Global spatial data were used for forest extent, global modelled biomass (GlobBiomass 97 and GeoCarbon 98 ), and soil carbon (GSOC 95 ) (S1.7). Carbon stock densities were extracted from the global maps for each site location, and for the spatial extent within the forest areas of each GEZ, to allow comparison of the modelled and measured data. The extant forest cover in Europe was differentiated into the mapped area of primary forest 28 (Supplementary Fig.  S4 ), and the remainder that was described as secondary forest, and thus had the potential to continue growing and accumulating carbon stocks.

First, the difference in carbon stock density (MgC ha −1 ) was estimated for primary forests between the site data and modelled values at the site locations and within the area of mapped primary forest 28 for each GEZ. Applying the ratio of this difference to the area of primary forest provided an updated estimate of the total carbon stock as the CCC of the extant primary forest (MgC). Second, the potential for carbon stocks to change in relation to natural ecological processes or human intervention was estimated from the difference between the current carbon stock (CCS) and the carbon carrying capacity (CCC). CCS of the secondary forest area within each GEZ was represented by the global modelled biomass. CCC was represented by the average carbon stock density (MgC ha −1 ) measured at primary forest sites within each GEZ and used to calculate the ratio of site data to global spatial data. This ratio was applied to convert the global spatial biomass in secondary forest for each GEZ to the equivalent of CCC. Results reported in Table  2 are for the sum of GEZs within each biome (boreal, temperate, subtropical). The workflow is described in S1.8 and details of the data in S2.3. The estimated potential carbon stock gain is only for the existing forest area and does not include the area that has been cleared of forest, that is, the potential natural forest vegetation.

Data availability

Data are available in the Supplementary Information, a Supplementary Data file and online at https://doi.org/10.6084/m9.figshare.25632561 with summaries for all datasets and the data for results presented. Data for individual research sites would need to be requested from country data custodians.

UNFCCC Paris Agreement. https://unfccc.int/sites/default/files/english_paris_agreement.pdf (2015).

UNFCCC Glasgow Climate Pact. https://unfccc.int/sites/default/files/resource/cop26_auv_2f_cover_decision.pdf (2021).

IPCC Global Warming of 1.5 °C: IPCC Special Report on Impacts of Global Warming of 1.5 °C above Pre-industrial Levels in Context of Strengthening Response to Climate Change, Sustainable Development, and Efforts to Eradicate Poverty. Cambridge University Press. https://doi.org/10.1017/9781009157940 (2022).

Mackey, B. G., et al. Untangling the confusion around land carbon science and climate change mitigation policy. Nat. Clim. Change 3 , https://doi.org/10.1038/NCLIMATE1804 (2013).

Hudiburg, T. W. et al. Meeting GHG reduction targets requires accounting for all forest sector emissions. Environ. Res. Lett. 14 , 095005 (2019).

Article   CAS   Google Scholar  

Carton, W., Lund, J. F. & Dooley, K. Undoing equivalence: rethinking carbon accounting for just carbon removal. Front. Clim. 3 , 664130 (2021).

Article   Google Scholar  

Keith, H. et al. Evaluating nature-based solutions for climate mitigation and conservation requires comprehensive carbon accounting. Sci. Total Environ. 769 , 144341 (2021).

Peng, L., Searchinger, T. D., Zionts, J. & Waite, R. The carbon costs of global wood harvests. Nature https://doi.org/10.1038/s41586-023-06187-1 (2023).

FAO Global Forest Resources Assessment 2020. Main Report. Rome. https://doi.org/10.4060/ca9825en (2020a).

Pan, Y. et al. A large and persistent carbon sink in the world’s forests. Science 333 , 988–993 (2011).

Garbarino, M. & Weisberg, P. J. Land-use legacies and forest change. Landsc. Ecol. 35 , 2641–2644 (2020).

Martin-Benito, D., Pederson, N., Férriz, M. & Gea-Izquierdo, G. Old forests and old carbon: a case study on the stand dynamics and longevity of aboveground carbon. Sci. Total Environ. 765 , 142737 (2021).

Grassi, G. et al. The key role of forests in meeting climate targets requires science for credible mitigation. Nat. Clim. Change 7 , 220–226 (2017).

EU Regulation of the European Parliament and of the Council amending Regulation (EU) 2018/841 as regards the scope, simplifying the compliance rules and setting out the targets of the Member States for 2030, and (EU) 2018/1999 as regards improvement in monitoring, reporting, tracking of progress and review. https://data.consilium.europa.eu/doc/document/ST-10774-2022-INIT/x/pdf (2022).

Grassi, G., Pilli, R., House, J., Federici, S. & Kurz, W. A. Science-based approach for credible accounting of mitigation in managed forests. Carbon Balance Manag. 13 , 8 (2018).

Keith, H., Mackey, B. G., Berry, S., Lindenmayer, D. B. & Gibbons, P. Estimating carbon carrying capacity in natural forest ecosystems across heterogeneous landscapes: addressing sources of error. Global Change Biol. 16 , 2971–2989 (2010).

Mikoláš, M., et al. Natural disturbance impacts on trade-offs and co-benefits of forest biodiversity and carbon. Proc. R. Soc. B 288 , 20211631 https://doi.org/10.1098/rspb.2021.1631 (2021).

FAO. Global Forest Resource Assessment 2020 . Terms and definitions (FAO, 2020b).

Buchwald, E. A. hierarchical terminology for more or less natural forests in relation to sustainable management and biodiversity conservation. In Proceedings: Third expert meeting on harmonizing forest-related definitions for use by various stakeholders (Food and Agriculture Organization of the United Nations, 2005).

Sabatini, F. M. et al. Where are Europe’s last primary forests? Divers. Distrib. 2018 , 1–14 (2017).

Google Scholar  

Vandekerkhove, K., et al. Old-growth criteria and indicators for beech forests ( Fageta ). LIFE_PROGNOSES Work Package 1.11. https://purews.inbo.be/ws/portalfiles/portal/74600061/Criteria_oldgrowth_PROGNOSES_Finalversion.pdf (2021).

Rogers, B. M., et al. Using ecosystem integrity to maximize climate mitigation and minimize risk in international forest policy. Front. Forests Global Change https://doi.org/10.3389/ffgc.2022.929281 (2022).

Stegen, J. C. et al. Variation in aboveground forest biomass across broad climatic gradients. Global Ecol. Biogeogr. 20 , 744–754 (2011).

Bastin, J.-F. et al. Pan-tropical prediction of forest structure from the largest trees. Global Ecol. Biogeogr. 27 , 1366–1383 (2018).

Erb, K.-H. et al. Unexpectedly large impact of forest management and grazing on global vegetation biomass. Nature 553 , 73–76 (2018).

Lutz, J. A. et al. Global importance of large-diameter trees. Global Ecol. Biogeogr. 8 , 1–16 (2018).

FAO. Global Ecological Zones for FAO forest reporting: 2010 update . Forest Resource Assessment Working Paper 179, Rome (FAO, 2012).

Sabatini, F. M. et al. European primary forest database v2.0. Sci. Data 8 , 220 (2021).

IPCC Refinement to the 2006 IPCC guidelines for national greenhouse gas inventories. https://www.ipcc.ch/report/2019-refinement-to-the-2006-ipcc-guidelines-for-national-greenhouse-gas-inventories/ (2019).

Eggleston, H. S., et al. (eds) IPCC Guidelines for National Greenhouse Gas Inventories, prepared by the National Greenhouse Gas Inventories Programme (IGES, Japan, 2006).

Nord-Larsen, T., Vesterdal, L., Bentsen, N. S. & Larsen, J. B. Ecosystem carbon stocks and their temporal resilience in a semi-natural beech-dominated forest. Forest Ecol. Manag. 447 , 67–76 (2019).

Santoro, M. et al. The global forest above-ground biomass pool for 2010 estimated from high-resolution satellite observations. Earth Syst. Sci. Data 13 , 3927–3950 (2021).

Rozendaal, D. M. A., Santoro, M., Schepaschenko, D., Avitabile, V., Herold, M. GlobBiomass Validation Report D17 v06. Prepared for European Space Agency (ESA-ESRIN). Friedrich Schiller University of Jena, Department of Earth Observation, Germany. https://www.dropbox.com/s/hplshh3ipm7hrak/GlobBiomass_D_17_Validation_Report_V06.pdf?dl=0 (2017).

Král, K. et al. Local variability of stand structural features in beech dominated natural forests on Central Europe: implications for sampling. For. Ecol. Manage 260 , 2196–2203 (2010).

Král, K., Valtera, M., Janík, D., Šamonil, P. & Vrška, T. Spatial variability of general stand characteristics in central European beech-dominated natural stands – effects of scale. For. Ecol. Manag. 328 , 353–364 (2014).

Martin-Benito, D., Molina-Valero, J. A., Pérez-Cruzado, C., Bigler, C. & Bugmann, H. Development and long-term dynamics of old-growth beech-fir forests in the Pyrenees: Evidence from dendroecology and dynamic vegetation modelling. For. Ecol. Manag. 524 , 120541 (2022).

Mikoláš, M. et al. Primary forest distribution and representation in a Central European landscape: Results of a large-scale field-based census. For. Ecol. Manag. 449 , 117466 (2019).

Holeksa, J. et al. A giant tree stand in the West Carpathians – an exception or a relic of formerly widespread mountain European forests? For. Ecol. Manag. 257 , 1577–1585 (2009).

Matuszkiewicz, J. M. et al. Current and potential carbon stock in the forest communities of the Bialowieza Biosphere Reserve. Forest Ecol. Manag. 502 , 119702 (2021).

Schulze, E.-D., et al. Temperate and Boreal Old-Growth Forests: How do Their Growth Dynamics and Biodiversity Differ from Young Stands and Managed Forests? In Old-Growth Forests: Function, Fate and Value , (eds, Wirth, C., Gleixner, G., Heimann, M.) 343–366, Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-92706-8_1 (2009).

Calders, K. et al. Laser scanning reveals potential underestimation of biomass carbon in temperate forest. Ecol. Solutions Evid. 3 , e12197 (2021).

Calders, K. et al. Nondestructive estimates of above-ground biomass using terrestrial laser scanning. Methods Ecol. Evol. 6 , 198–208 (2015).

Keith, H., et al. The Land Gap Report . https://www.landgap.org/ (2022).

Ķēniņa, L., Elferts, D., Jaunslaviete, I., Bāders, E., Jansons, Ā. Sustaining carbon storage: lessons from hemiboreal old-growth coniferous and deciduous forest stands. Forest Science 1–9, https://doi.org/10.1093/forsci/fxac055 (2022a).

Bastin, J.-F. et al. Seeing central African forests through their largest trees. Scientific Reports 5 , 13156 (2015).

Slik, J. W. F. et al. Large trees drive forest aboveground biomass variation in moist lowland forests across the tropics. Glob. Ecol. Biogeogr. 22 , 1261–1271 (2013).

Ķēniņa, L. et al. Tree biomass – a fragile carbon storage in old-growth birch and aspen stands in hemiboreal Latvia. Baltic Forestry 28 , 654 (2022b).

Senf, C. & Seidl, R. Mapping the forest disturbance regimes of Europe. Nat. Sustain. 4 , 63–70 (2020).

Ceccherini, G. et al. Abrupt increase in harvested forest area over Europe after 2015. Nature 583 , 72–77 (2020).

Ceccherini, G. et al. Reply to Wernick IK et al.; Palahi M et al. 2021. Nature 592 , E18–E23 (2021).

Palahi, M. et al. Concerns about reported harvests in European forests. Nature 592 , E15–E17 (2021).

Wernick, I. K. et al. Quantifying forest change in the European Union. Nature 592 , E13–E14 (2021).

Camia, A. et al. The use of woody biomass for energy production in the EU and impacts on forests. JRC Science for Policy Report EUR 30548 EN (Publications Office of the European Union, 2020). https://doi.org/10.2760/831621 (2020).

Senf, C. et al. Canopy mortality has doubled in Europe’s temperate forests over the last three decades. Nat. Commun. 9 , 4978 (2018).

Seidl, R., Schelhaas, M. J., Rammer, W. & Verkerk, P. J. Increasing forest disturbances in Europe and their impact on carbon storage. Nat. Clim. Change 4 , 806–810 (2014).

Keith, H. et al. Accounting for biomass carbon stock change due to wildfire in temperate forest landscapes in Australia. Public Library Sci. PLoS One 9 , e10712 (2014).

Thompson, I., Mackey, B., McNulty, S., Mosseler, A. Forest Resilience, Biodiversity, and Climate Change. A synthesis of the biodiversity/resilience/stability relationship in forest ecosystems. Secretariat of the Convention on Biological Diversity, Montreal. Technical Series no. 43. p. 67. https://doi.org/10.1111/ele.12054 (2009).

Alberto, F. et al. Potential for evolutionary responses to climate change – evidence from tree populations. Global Change Biol. 19 , 1645–1661 (2013).

Barber, C. V., Petersen, R., Young, V., Mackey, B., Kormos, C. The Nexus Report: Nature Based Solutions to the Biodiversity and Climate Crisis. F20 Foundations, Campaign for Nature and SEE Foundation. https://foundations-20.org/publication/the-nexus-report-nature-based-solutions-to-the-biodiversity-and-climate-crisis/ (2020).

Di Filippo, A., Biondi, F., Piovesan, G. & Ziaco, E. Tree ring‐based metrics for assessing old‐growth forest naturalness. J. Appl. Ecol. 54 , 737–749 (2017).

Carey, E. V., Sala, A., Keane, R. & Callaway, R. M. Are old forests underestimated as global carbon sinks. Glob. Chang. Biol. 7 , 339–344 (2001).

Zhou, G. et al. Old-Growth Forests Can Accumulate Carbon in Soils. Science 314 , 1417–1417 (2006).

Glatthorn, J., Feldmann, E., Pichler, V., Hauck, M. & Leuschner, C. Biomass Stock and Productivity of Primeval and Production Beech Forests: Greater Canopy Structural Diversity Promotes Productivity. Ecosystems 21 , 704–722 (2018).

Campioli, M. et al. Biomass production efficiency controlled by management in temperate and boreal ecosystems. Nat. Geosci. 8 , 843–846 (2015).

Knohl, A., Schulze, E.-D., Kolle, O. & Buchmann, N. Large carbon uptake by an unmanaged 250-year-old deciduous forest in Central Germany. Agric. For. Meteorol. 118 , 151–167 (2003).

Gundersen, P. et al. Old-growth forest carbon sinks overestimated. Nature 591 , E21–E23 (2021).

Luyssaert, S. et al. Reply to: Old-growth forest carbon sinks overestimated. Nature 591 , E24–E25 (2021).

Stephenson, N. L. et al. Rate of tree carbon accumulation increases continuously with tree size. Nature 507 , 90–93 (2014).

Gough, C. M., Curtis, P. S., Hardiman, B. S., Scheuermann, C. M. & Bond-Lamberty, B. Disturbance, complexity, and succession of net ecosystem production in North America’s temperate deciduous forests. Ecosphere 7 , e01375 (2016).

Vandekerkhove, K. et al. Very large trees in a lowland old-growth beech (Fagus sylvatica L.) forest: density, size, growth and spatial patterns in comparison to reference sites in Europe. Forest Ecol. Manag. 417 , 1–17 (2018).

Birdsey, R. A., et al. Assessing carbon stocks and accumulation potential of mature forests and larger trees in U.S. federal lands. Frontiers in Forests and Global Change . https://doi.org/10.3389/ffgc.2022.1074508 (2023).

Achat, D. L., Fortin, M., Landmann, G., Ringeval, B. & Augusto, L. Forest soil carbon is threatened by intensive biomass harvesting. Sci. Rep. 5 , 15991 (2015).

Mayer, M. et al. Influence of forest management activities on soil organic carbon stocks: a knowledge synthesis. For. Ecol. Manage. 466 , 118127 (2020).

Harris, N. et al. Global maps of twenty-first century forest carbon fluxes. Nat. Clim. Change 11 , 234–240 (2021).

Bellassen, V. & Luyssaert, S. Managing forests in uncertain times. Nature 506 , 153–155 (2014).

Houghton, R. A., Byers, B. & Nassikas, A. A. A role for tropical forests in stabilizing atmospheric CO 2 . Nat. Clim. Change 5 , 1022–1023, (2015).

Keith, H., Lindenmayer, D., Macintosh, A. & Mackey, B. Under What Circumstances Do Wood Products from Native Forests Benefit Climate Change Mitigation? Plos One 10 , e0139640 (2015).

Meyer, P., Nagel, R. & Feldmann, E. Limited sink but large storage: Biomass dynamics in naturally developing beech ( Fagus sylvatica ) and oak ( Quercus robur, Quercus petraea ) forests of north-western Germany. J. Ecol. 2021 , 1–15 (2021). 00 .

Idoate-Lacasia, J. et al. Long-term biomass dynamics of temperate forests in Europe after cessation of management. Forest Ecol. Manag. 554 , 121697 (2024).

Naudts, K. et al. Europe’s forest management did not mitigate climate warming. Science 351 , 597–600 (2016).

Barredo, J. I. Mapping and assessment of primary and old-growth forests in Europe. JRC Science for Policy Report, European Commission. EUR 30661 EN, Publications Office of the European Union, https://doi.org/10.2760/797591 (2021).

Burrascano, S., Keeton, W. S., Sabatini, F. M. & Blasi, C. Commonality and variability in the structural attributes of moist temperate old-growth forests: A global review. Forest Ecol. Manag. 291 , 458–479 (2013).

Eggers, T. The impacts of manufacturing and utilisation of wood products on the European carbon budget. European Forest Institute, Internal Report 9. https://efi.int/sites/default/files/files/publication-bank/2018/ir_09.pdf (2002).

Brunet-Navarro, P., Jochheim, H. & Muys, B. The effect of increasing lifespan and recycling rate on carbon storage in wood products from theoretical model to application for the European wood sector. Mitig Adapt Strateg Glob Change 22 , 1193–1205 (2017).

European Commission. The European Commission’s Knowledge Centre for Bioeconomy. Brief on the role of the forest-based bioeconomy in mitigating climate change through carbon storage and material substitution. https://knowledge4policy.ec.europa.eu/bioeconomy/topic/forest-bioeconomy-cc-mitigation_en (2022).

Ciais, P. et al. Carbon accumulation in European forests. Nat. Geosci. 1 , 425–429 (2008).

Adams, J. M. & Piovesan, G. Uncertainties in the role of land vegetation in the carbon cycle. Chemosphere 49 , 805–819 (2002).

Korosuo, A. et al. The role of forests in the EU climate policy: are we on the right track? Carbon Balance Manag. 18 , 15 (2023).

UNFCCC 2023. European Union Common Reporting Format Table https://unfccc.int/documents/627830 (2023).

European Commission Biodiversity Strategy for 2030. https://environment.ec.europa.eu/strategy/biodiversity-strategy-2030_en (2023).

Sabatini, F. M. et al. Protection gaps and restoration opportunities for primary forests in Europe. Divers. Distrib. 26 , 1646–1662 (2020).

Maes, J., et al. Mapping and assessment of ecosystems and their services: an EU ecosystem assessment. JRC Science for Policy Report. EUR 30161, Publications Office of the European Union. https://doi.org/10.2760/757183 (2020).

Thomas, C. D. et al. Reconciling biodiversity and carbon conservation. Ecol. Lett. 16 , 39–47 (2013).

European Environment Agency, CORINE Land Cover https://land.copernicus.eu/pan-european/corine-land-cover (2018).

FAO Global Soil Organic Carbon GSOC map v 1.5. Technical Report, Intergovernmental Technical Panel on Soils, Rome FAO. https://www.fao.org/documents/card/en/c/ca7597en . https://www.fao.org/global-soil-partnership/pillars-action/4-information-and-data-new/global-soil-organic-carbon-gsoc-map/en/ (2020c).

Bottero, A., Meloni, F., Garbarino, M. & Motta, R. Temperate coppice forests in north-western Italy are resilient to wild ungulate browsing in the short to medium term. For Ecol Manage 523 , 120484 (2022).

Santoro, M., et al. GlobBiomass: global above-ground biomass and growing stock volume datasets available on-line at http://globbiomass.org/products/global-mapping (2018).

Avitabile, V. et al. An integrated pan-tropical biomass map using multiple reference datasets. Global Change Biol. 22 , 1406–1420 (2016).

Forest Europe State of Europe’s Forests 2020. https://foresteurope.org/wp-content/uploads/2016/08/SoEF_2020.pdf (2020).

Download references

Acknowledgements

We thank the many people involved with the collection and provision of the site data and recognise the significant resources, people and time required to collect this invaluable data. The research for the synthesis, analysis and writing (H.K., Z.K., S.H., B.M.) was supported by a grant from a charitable organisation which neither seeks nor permits publicity for its efforts. The funder had no involvement in the study design, results or publication of the paper. Site data from Spain was funded by the Spanish Ministry of Science, Innovation and Universities [AGL2016-76769-C2-2-R]. C.P.C. was supported by the Spanish Ministry of Science and Innovation [RYC2018-024939-I]. J.A.M.V. was supported by the Ramón Areces Foundation Grants for Postdoctoral Studies. Contribution of D.A., K.K. and P.S. as well as data collection and processing from Czech natural forests was supported by Czech Science Foundation, project no. 24-11119S. D.M.-B. was funded by projects AGL2015-73190-JIN, PID2019-110273RB-I00 and contract RYC-2017-23389 by the Spanish Ministry of Science and Innovation MCIN/AEI. V.B. and I.D. were supported by the FORCLIMIT project funded in the frame of the ERA-NET FACCE ERA-GAS and with national support from Romanian National Authority for Scientific Research and Innovation, CCCDI – UEFISCDI [grant number 82/2017]. FACCE ERA-GAS has received funding from the European Union’s Horizon 2020 research and innovation programme [grant agreement 696356. T.Z. was funded by The WWF Bulgaria through the project IKEA № 9E0710.05 and by The National Roadmap for Research Infrastructure (2020-2027), Ministry of Education and Science of Republic of Bulgaria, through agreements No DO1-405/18.12.2020 and DO1-163/28.07.2022 (LTER-BG). L.D. was funded by the project of the National Research, Development and Innovation Office NKFIH K 131837. T.N. received support from the Slovenian Research Agency (Project No. J4-1765). For additional assistance with site data, we thank Dr. Raúl Sanchez-Salguero and Dr. Andrea Hevia for evaluating the age in the dendrochronological samples in Spain, and Nesibe Köse, Mehmet Doğan, Daniel Bishop, Marco Mina, Timothy Thrippleton, Neil Pederson, Guillermo Gea-Izquierdo and Macarena Férriz for their help during fieldwork, and Cengiz Cihan and the Turkish General Directorate of Forestry (OGM) in Borçka (Artvin) for their assistance in the field in Turkey.

Author information

László Demeter

Present address: School of Geography, University of Nottingham, University Park, Nottingham, NG7 2RD, United Kingdom

Authors and Affiliations

Climate Action Beacon, Griffith University, Parklands Drive, Southport, QLD, 4222, Australia

Heather Keith, Sonia Hugh & Brendan Mackey

Wild Europe, Györ, 9011, Hungary

Czech University of Life Sciences Prague, Faculty of Forestry and Wood Sciences, Kamýcká 129, Suchdol, 16 521, Praha 6, Czech Republic

Miroslav Svoboda, Martin Mikoláš, Juan Alberto Molina-Valero, Thomas A. Nagel & Ondřej Vostarek

Department of Forest Ecology, Silva Tarouca Research Institute (VÚKOZ), Lidicka 25/27, 602 00, Brno, Czech Republic

Dusan Adam, Kamil Kral & Pavel Šamonil

Scientific Department, Belovezhskaya Pushcha National Park, Kameniuki, Belarus

Dmitry Bernatski

Department of Silviculture, Transilvania University of Brasov, Sirul Beethoven, 1, Brasov, Romania

Viorel Blujdea & Ioan Dutcă

Joint Research Centre, D.1 Forests and Bioeconomy Unit, Via E. Fermi 2749-TP 261, I-21027, Ispra, Italy

Viorel Blujdea

Department of Ecological Modelling, Helmholz Centre for Environmental Research UFZ, Permoserstraße 15, 04318, Leipzig, Germany

Friedrich Bohn

Instituto Pirenaico de Ecología (IPE-CSIC), Avda. Montañana, 1005, 50192, Zaragoza, Spain

Jesús Julio Camarero

Institute of Ecology and Botany, HUN-REN Centre for Ecological Research, Alkotmány, Hungary

László Demeter & Ferenc Horváth

Department of Agricultural, Forest and Food Sciences, Dendro Lab, Università della Tuscia, Viterbo, Italy

Alfredo Di Filippo

Buckinghamshire New University, Queen Alexandra Road, High Wycombe, HP11 2JZ, United Kingdom

Department of Agricultural, Forest and Food Sciences, University of Torino, Largo Paolo Braccini 2-10095, Grugliasco, Italy

Matteo Garbarino & Renzo Motta

Scientific Department, Berezinski Biosphere Reserve, Domzheritsy, Belarus

Valery Ivkovich

Latvian State Forest Research Institute Silava, Rigas Street 111, LV-2169, Salaspils, Latvia

Āris Jansons & Laura Ķēņina

Institute of Forest Sciences, ICIFOR, INIA-CSIC, Ctra. La Coruña km. 7.5, 28040, Madrid, Spain

Dario Martin-Benito

Department of Forestry and Renewable Forest Resources, Biotechnical Faculty, University of Ljubljana, Ljubljana, Slovenia

Thomas A. Nagel

Dendrology Department, University of Forestry, Sofia, Bulgaria

Momchil Panayotov

Higher Polytechnic School of Engineering, University of Santiago de Compostela, Benigno Ledo s/n E-27002, Lugo, Spain

César Pérez-Cruzado

Department of Ecological and Biological Sciences, Landscape Ecological Forest Planning Lab, Università della Tuscia, Viterbo, Italy

Gianluca Piovesan

Forest Biometrics Laboratory – Faculty of Forestry, Stefan cel Mare University of Suceava, Universității Street number.13, Suceava, 720229, Romania

Cătălin-Constantin Roibu

Institute of Experimental Botany, National Academy of Science of Belarus, Minsk, 220072, Academicheskaya St. 27, Belarus

Maxim Yermokhin

Institute of Biodiversity and Ecosystem Research, Bulgarian Academy of Sciences, Sofia, Bulgaria

Tzvetan Zlatanov

You can also search for this author in PubMed   Google Scholar

Contributions

H.K., Z.K., B.M. conceptualised the study; Z.K. coordinated data providers; H.K. and S.H. analysed the data; H.K. wrote the paper; M.S., M.M., D.A., D.B., V.B., F.B., J.J.C., L.D., A.D.F., I.D., M.G., A.J., F.H., V.I., L.K., K.K., D.M.B., J.A.M.V., R.M., T.A.N., M.P., C.P.C., G.P., C.C.R., P.S., O.V., M.Y., T.Z. contributed site data and interpretation, all authors contributed to editing and review of the manuscript.

Corresponding author

Correspondence to Heather Keith .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Communications Earth & Environment thanks Per Gundersen and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editors: Alireza Bahadori and Aliénor Lavergne. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Peer review file, supplementary information, description of additional supplementary files, supplementary data 1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Keith, H., Kun, Z., Hugh, S. et al. Carbon carrying capacity in primary forests shows potential for mitigation achieving the European Green Deal 2030 target. Commun Earth Environ 5 , 256 (2024). https://doi.org/10.1038/s43247-024-01416-5

Download citation

Received : 20 April 2023

Accepted : 23 April 2024

Published : 14 May 2024

DOI : https://doi.org/10.1038/s43247-024-01416-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

This paper is in the following e-collection/theme issue:

Published on 13.5.2024 in Vol 26 (2024)

Suitability of the Current Health Technology Assessment of Innovative Artificial Intelligence-Based Medical Devices: Scoping Literature Review

Authors of this article:

There are no citations yet available for this article according to Crossref .