development method of research

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Best Family Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Verywell Mind Insights
• 2023 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

Developmental Psychology Research Methods

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Emily is a board-certified science editor who has worked with top digital publishing brands like Voices for Biodiversity, Study.com, GoodTherapy, Vox, and Verywell.

Jose Luis Pelaez Inc/Getty Images 

Cross-Sectional Research Methods

Longitudinal research methods, correlational research methods, experimental research methods.

There are many different developmental psychology research methods, including cross-sectional, longitudinal, correlational, and experimental. Each has its own specific advantages and disadvantages. The one that a scientist chooses depends largely on the aim of the study and the nature of the phenomenon being studied.

Research design provides a standardized framework to test a hypothesis and evaluate whether the hypothesis is correct, incorrect, or inconclusive. Even if the hypothesis is untrue, the research can often provide insights that may prove valuable or move research in an entirely new direction.

At a Glance

In order to study developmental psychology, researchers utilize a number of different research methods. Some involve looking at different cross-sections of a population, while others look at how participants change over time. In other cases, researchers look at how whether certain variables appear to have a relationship with one another. In order to determine if there is a cause-and-effect relationship, however, psychologists much conduct experimental research.

Learn more about each of these different types of developmental psychology research methods, including when they are used and what they can reveal about human development.

Cross-sectional research involves looking at different groups of people with specific characteristics.

For example, a researcher might evaluate a group of young adults and compare the corresponding data from a group of older adults.

The benefit of this type of research is that it can be done relatively quickly; the research data is gathered at the same point in time. The disadvantage is that the research aims to make a direct association between a cause and an effect. This is not always so easy. In some cases, there may be confounding factors that contribute to the effect.

To this end, a cross-sectional study can suggest the odds of an effect occurring both in terms of the absolute risk (the odds of something happening over a period of time) and the relative risk (the odds of something happening in one group compared to another).  

Longitudinal research involves studying the same group of individuals over an extended period of time.

Data is collected at the outset of the study and gathered repeatedly through the course of study. In some cases, longitudinal studies can last for several decades or be open-ended. One such example is the Terman Study of the Gifted , which began in the 1920s and followed 1528 children for over 80 years.

The benefit of this longitudinal research is that it allows researchers to look at changes over time. By contrast, one of the obvious disadvantages is cost. Because of the expense of a long-term study, they tend to be confined to a smaller group of subjects or a narrower field of observation.

Challenges of Longitudinal Research

While revealing, longitudinal studies present a few challenges that make them more difficult to use when studying developmental psychology and other topics.

• Longitudinal studies are difficult to apply to a larger population.
• Another problem is that the participants can often drop out mid-study, shrinking the sample size and relative conclusions.
• Moreover, if certain outside forces change during the course of the study (including economics, politics, and science), they can influence the outcomes in a way that significantly skews the results.

For example, in Lewis Terman's longitudinal study, the correlation between IQ and achievement was blunted by such confounding forces as the Great Depression and World War II (which limited educational attainment) and gender politics of the 1940s and 1950s (which limited a woman's professional prospects).

Correlational research aims to determine if one variable has a measurable association with another.

In this type of non-experimental study, researchers look at relationships between the two variables but do not introduce the variables themselves. Instead, they gather and evaluate the available data and offer a statistical conclusion.

For example, the researchers may look at whether academic success in elementary school leads to better-paying jobs in the future. While the researchers can collect and evaluate the data, they do not manipulate any of the variables in question.

A correlational study can be appropriate and helpful if you cannot manipulate a variable because it is impossible, impractical, or unethical.

For example, imagine that a researcher wants to determine if living in a noisy environment makes people less efficient in the workplace. It would be impractical and unreasonable to artificially inflate the noise level in a working environment. Instead, researchers might collect data and then look for correlations between the variables of interest.

Limitations of Correlational Research

Correlational research has its limitations. While it can identify an association, it does not necessarily suggest a cause for the effect. Just because two variables have a relationship does not mean that changes in one will affect a change in the other.

Unlike correlational research, experimentation involves both the manipulation and measurement of variables . This model of research is the most scientifically conclusive and commonly used in medicine, chemistry, psychology, biology, and sociology.

Experimental research uses manipulation to understand cause and effect in a sampling of subjects. The sample is comprised of two groups: an experimental group in whom the variable (such as a drug or treatment) is introduced and a control group in whom the variable is not introduced.

Deciding the sample groups can be done in a number of ways:

• Population sampling, in which the subjects represent a specific population
• Random selection , in which subjects are chosen randomly to see if the effects of the variable are consistently achieved

Challenges in Experimental Resarch

While the statistical value of an experimental study is robust, it may be affected by confirmation bias . This is when the investigator's desire to publish or achieve an unambiguous result can skew the interpretations, leading to a false-positive conclusion.

One way to avoid this is to conduct a double-blind study in which neither the participants nor researchers are aware of which group is the control. A double-blind randomized controlled trial (RCT) is considered the gold standard of research.

What This Means For You

There are many different types of research methods that scientists use to study developmental psychology and other areas. Knowing more about how each of these methods works can give you a better understanding of what the findings of psychological research might mean for you.

Capili B. Cross-sectional studies .  Am J Nurs . 2021;121(10):59-62. doi:10.1097/01.NAJ.0000794280.73744.fe

Kesmodel US. Cross-sectional studies - what are they good for? .  Acta Obstet Gynecol Scand . 2018;97(4):388–393. doi:10.1111/aogs.13331

Noordzij M, van Diepen M, Caskey FC, Jager KJ. Relative risk versus absolute risk: One cannot be interpreted without the other . Nephrology Dialysis Transplantation. 2017;32(S2):ii13-ii18. doi:10.1093/ndt/gfw465

Kell HJ, Wai J. Terman Study of the Gifted . In: Frey B, ed.  The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation . Vol. 4. Thousand Oaks, CA: SAGE Publications, Inc.; 2018. doi:10.4135/9781506326139.n691

Curtis EA, Comiskey C, Dempsey O. Importance and use of correlational research .  Nurse Res . 2016;23(6):20–25. doi:10.7748/nr.2016.e1382

Misra S.  Randomized double blind placebo control studies, the "Gold Standard" in intervention based studies .  Indian J Sex Transm Dis AIDS . 2012;33(2):131-4. doi:10.4103/2589-0557.102130

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons
• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

6.1: Research Methods in Developmental Psychology

• Last updated
• Save as PDF
• Page ID 10627

• https://nobaproject.com/ via The Noba Project

University of Calfornia, Irvine

What do infants know about the world in which they live – and how do they grow and change with age? These are the kinds of questions answered by developmental scientists. This module describes different research techniques that are used to study psychological phenomena in infants and children, research designs that are used to examine age-related changes in development, and unique challenges and special issues associated with conducting research with infants and children. Child development is a fascinating field of study, and many interesting questions remain to be examined by future generations of developmental scientists – maybe you will be among them!

learning objectives

• Describe different research methods used to study infant and child development
• Discuss different research designs, as well as their strengths and limitations
• Report on the unique challenges associated with conducting developmental research

Introduction

A group of children were playing hide-and-seek in the yard. Pilar raced to her hiding spot as her six-year-old cousin, Lucas, loudly counted, “… six, seven, eight, nine, ten! Ready or not, here I come!”. Pilar let out a small giggle as Lucas ran over to find her – in the exact location where he had found his sister a short time before. At first glance, this behavior is puzzling: why would Pilar hide in exactly the same location where someone else was just found? Whereas older children and adults realize that it is likely best to hide in locations that have not been searched previously, young children do not have the same cognitive sophistication. But why not… and when do these abilities first develop?

Developmental psychologists investigate questions like these using research methods that are tailored to the particular capabilities of the infants and children being studied. Importantly, research in developmental psychology is more than simply examining how children behave during games of hide-and-seek – the results obtained from developmental research have been used to inform best practices in parenting, education, and policy.

This module describes different research techniques that are used to study psychological phenomena in infants and children, research designs that are used to examine age-related changes in developmental processes and changes over time, and unique challenges and special issues associated with conducting research with infants and children.

Research Methods

Infants and children—especially younger children—cannot be studied using the same research methods used in studies with adults. Researchers, therefore, have developed many creative ways to collect information about infant and child development. In this section, we highlight some of the methods that have been used by researchers who study infants and older children, separating them into three distinct categories: involuntary or obligatory responses , voluntary responses , and psychophysiological responses . We will also discuss other methods such as the use of surveys and questionnaires. At the end of this section, we give an example of how interview techniques can be used to study the beliefs and perceptions of older children and adults – a method that cannot be used with infants or very young children.

Involuntary or obligatory responses

One of the primary challenges in studying very young infants is that they have limited motor control – they cannot hold their heads up for short amounts of time, much less grab an interesting toy, play the piano, or turn a door knob. As a result, infants cannot actively engage with the environment in the same way as older children and adults. For this reason, developmental scientists have designed research methods that assess involuntary or obligatory responses. These are behaviors in which people engage without much conscious thought or effort. For example, think about the last time you heard your name at a party – you likely turned your head to see who was talking without even thinking about it. Infants and young children also demonstrate involuntary responses to stimuli in the environment. When infants hear the voice of their mother, for instance, their heart rate increases – whereas if they hear the voice of a stranger, their heart rate decreases (Kisilevsky et al., 2003). Researchers study involuntary behaviors to better understand what infants know about the world around them.

One research method that capitalizes on involuntary or obligatory responses is a procedure known as habituation . In habituation studies, infants are presented with a stimulus such as a photograph of a face over and over again until they become bored with it. When infants become bored, they look away from the picture. If infants are then shown a new picture--such as a photograph of a different face-- their interest returns and they look at the new picture. This is a phenomenon known as dishabituation . Habituation procedures work because infants generally look longer at novel stimuli relative to items that are familiar to them. This research technique takes advantage of involuntary or obligatory responses because infants are constantly looking around and observing their environments; they do not have to be taught to engage with the world in this way.

One classic habituation study was conducted by Baillargeon and colleagues (1985). These researchers were interested in the concept of object permanence , or the understanding that objects exist even when they cannot be seen or heard. For example, you know your toothbrush exists even though you are probably not able to see it right this second. To investigate object permanence in 5-month-old infants, the researchers used a violation of expectation paradigm . The researchers first habituated infants to an opaque screen that moved back and forth like a drawbridge (using the same procedure you just learned about in the previous paragraph). Once the infants were bored with the moving screen, they were shown two different scenarios to test their understanding of physical events. In both of these test scenarios, an opaque box was placed behind the moving screen. What differed between these two scenarios, however, was whether they confirmed or violated the solidity principle – the idea that two solid objects cannot occupy the same space at the same time. In the possible scenario, infants watched as the moving drawbridge stopped when it hit the opaque box (as would be expected based on the solidity principle). In the impossible scenario, the drawbridge appeared to move right through the space that was occupied by the opaque box! This impossible scenario violates the solidity principle in the same way as if you got out of your chair and walked through a wall, reappearing on the other side.

The results of this study revealed that infants looked longer at the impossible test event than at the possible test event. The authors suggested that the infants reacted in this way because they were surprised – the demonstration went against their expectation that two solids cannot move through one another. The findings indicated that 5-month-old infants understood that the box continued to exist even when they could not see it. Subsequent studies indicated that 3½- and 4½-month-old infants also demonstrate object permanence under similar test conditions (Baillargeon, 1987). These findings are notable because they suggest that infants understand object permanence much earlier than had been reported previously in research examining voluntary responses (although see more recent research by Cashon & Cohen, 2000).

Voluntary responses

As infants and children age, researchers are increasingly able to study their understanding of the world through their voluntary responses. Voluntary responses are behaviors that a person completes by choice. For example, think about how you act when you go to the grocery store: you select whether to use a shopping cart or a basket, you decide which sections of the store to walk through, and you choose whether to stick to your grocery list or splurge on a treat. Importantly, these behaviors are completely up to you (and are under your control). Although they do not do a lot of grocery shopping, infants and children also have voluntary control over their actions. Children, for instance, choose which toys to play with.

Researchers study the voluntary responses of infants and young children in many ways. For example, developmental scientists study recall memory in infants and young children by looking at voluntary responses. Recall memory is memory of past events or episodes, such as what you did yesterday afternoon or on your last birthday. Whereas older children and adults are simply asked to talk about their past experiences, recall memory has to be studied in a different way in infants and very young children who cannot discuss the past using language. To study memory in these subjects researchers use a behavioral method known as elicited imitation (Lukowski & Milojevich, in press).

In the elicited imitation procedure, infants play with toys that are designed in the lab to be unlike the kinds of things infants usually have at home. These toys (or event sequences, as researchers call them) can be put together in a certain way to produce an outcome that infants commonly enjoy. One of these events is called Find the Surprise. As shown in Figure 6.1.1, this toy has a door on the front that is held in place by a latch – and a small plastic figure is hidden on the inside. During the first part of the study, infants play with the toy in whichever way they want for a few minutes. The researcher then shows the infant how make the toy work by (1) flipping the latch out of the way and (2) opening the door, revealing the plastic toy inside. The infant is allowed to play with the toy again either immediately after the demonstration or after a longer delay. As the infant plays, the researcher records whether the infant finds the surprise using the same procedure that was demonstrated.

Use of the elicited imitation procedure has taught developmental scientists a lot about how recall memory develops. For example, we now know that 6-month-old infants remember one step of a 3-step sequence for 24 hours (Barr, Dowden, & Hayne, 1996; Collie & Hayne, 1999). Nine-month-olds remember the individual steps that make up a 2-step event sequence for 1 month, but only 50% of infants remember to do the first step of the sequence before the second (Bauer, Wiebe, Carver, Waters, & Nelson, 2003; Bauer, Wiebe, Waters, & Bangston, 2001; Carver & Bauer, 1999). When children are 20 months old, they remember the individual steps and temporal order of 4-step events for at least 12 months – the longest delay that has been tested to date (Bauer, Wenner, Dropik, & Wewerka, 2000).

Psychophysiology

Behavioral studies have taught us important information about what infants and children know about the world. Research on behavior alone, however, cannot tell scientists how brain development or biological changes impact (or are impacted by) behavior. For this reason, researchers may also record psychophysiological data, such as measures of heart rate, hormone levels, or brain activity. These measures may be recorded by themselves or in combination with behavioral data to better understand the bidirectional relations between biology and behavior.

One manner of understanding associations between brain development and behavioral advances is through the recording of event-related potentials , or ERPs. ERPs are recorded by fitting a research participant with a stretchy cap that contains many small sensors or electrodes. These electrodes record tiny electrical currents on the scalp of the participant in response to the presentation of particular stimuli, such as a picture or a sound (for additional information on recording ERPs from infants and children, see DeBoer, Scott, & Nelson, 2005). The recorded responses are then amplified thousands of times using specialized equipment so that they look like squiggly lines with peaks and valleys. Some of these brain responses have been linked to psychological phenomena. For example, researchers have identified a negative peak in the recorded waveform that they have called the N170 (Bentin, Allison, Puce, Perez, & McCarthy, 2010). The peak is named in this way because it is negative (hence the N) and because it occurs about 140ms to 170ms after a stimulus is presented (hence the 170). This peak is particularly sensitive to the presentation of faces, as it is commonly more negative when participants are presented with photographs of faces rather than with photographs of objects. In this way, researchers are able to identify brain activity associated with real world thinking and behavior.

The use of ERPs has provided important insight as to how infants and children understand the world around them. In one study (Webb, Dawson, Bernier, & Panagiotides, 2006), researchers examined face and object processing in children with autism spectrum disorders, those with developmental delays, and those who were typically developing. The children wore electrode caps and had their brain activity recorded as they watched still photographs of faces (of their mother or of a stranger) and objects (including those that were familiar or unfamiliar to them). The researchers examined differences in face and object processing by group by observing a component of the brainwave they called the prN170 (because it was believed to be a precursor to the adult N170). Their results showed that the height of the prN170 peak (commonly called the amplitude ) did not differ when faces or objects were presented to typically developing children. When considering children with autism, however, the peaks were higher when objects were presented relative to when faces were shown. Differences were also found in how long it took the brain to reach the negative peak (commonly called the latency of the response). Whereas the peak was reached more quickly when typically developing children were presented with faces relative to objects, the opposite was true for children with autism. These findings suggest that children with autism are in some way processing faces differently than typically developing children (and, as reported in the manuscript, children with more general developmental delays).

Parent-report questionnaires

Developmental science has come a long way in assessing various aspects of infant and child development through behavior and psychophysiology – and new advances are happening every day. In many ways, however, the very youngest of research participants are still quite limited in the information they can provide about their own development. As such, researchers often ask the people who know infants and children best – commonly, their parents or guardians – to complete surveys or questionnaires about various aspects of their lives. These parent-report data can be analyzed by themselves or in combination with any collected behavioral or psychophysiological data.

One commonly used parent-report questionnaire is the Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2000). Parents complete the preschooler version of this questionnaire by answering questions about child strengths, behavior problems, and disabilities, among other things (click here to see sample questions). The responses provided by parents are used to identify whether the child has any behavioral issues, such as sleep difficulties, aggressive behaviors, depression, or attention deficit/hyperactivity problems.

A recent study used the CBCL-Preschool questionnaire (Achenbach & Rescorla, 2000) to examine preschooler functioning in relation to levels of stress experienced by their mothers while they were pregnant (Ronald, Pennell, & Whitehouse, 2011). Almost 3,000 pregnant women were recruited into the study during their pregnancy and were interviewed about their stressful life experiences. Later, when their children were 2 years old, mothers completed the CBCL-Preschool questionnaire. The results of the study showed that higher levels of maternal stress during pregnancy (such as a divorce or moving to a new house) were associated with increased attention deficit/hyperactivity problems in children over 2 years later. These findings suggest that stressful events experienced during prenatal development may be associated with problematic child behavioral functioning years later – although additional research is needed.

Interview techniques

Whereas infants and very young children are unable to talk about their own thoughts and behaviors, older children and adults are commonly asked to use language to discuss their thoughts and knowledge about the world. In fact, these verbal report paradigms are among the most widely used in psychological research. For instance, a researcher might present a child with a vignette or short story describing a moral dilemma, and the child would be asked to give their own thoughts and beliefs (Walrath, 2011). For example, children might react to the following:

“Mr. Kohut’s wife is sick and only one medication can save her life. The medicine is extremely expensive and Mr. Kohut cannot afford it. The druggist will not lower the price. What should Mr. Kohut do, and why?”

Children can provide written or verbal answers to these types of scenarios. They can also offer their perspectives on issues ranging from attitudes towards drug use to the experience of fear while falling asleep to their memories of getting lost in public places – the possibilities are endless. Verbal reports such as interviews and surveys allow children to describe their own experience of the world.

Research Design

Now you know about some tools used to conduct research with infants and young children. Remember, research methods are the tools that are used to collect information. But it is easy to confuse research methods and research design . Research design is the strategy or blueprint for deciding how to collect and analyze information. Research design dictates which methods are used and how.

Researchers typically focus on two distinct types of comparisons when conducting research with infants and children. The first kind of comparison examines change within individuals . As the name suggests, this type of analysis measures the ways in which a specific person changes (or remains the same) over time. For example, a developmental scientist might be interested in studying the same group of infants at 12 months, 18 months, and 24 months to examine how vocabulary and grammar change over time. This kind of question would be best answered using a longitudinal research design. Another sort of comparison focuses on changes between groups . In this type of analysis, researchers study average changes in behavior between groups of different ages. Returning to the language example, a scientist might study the vocabulary and grammar used by 12-month-olds, 18-month-olds, and 24-month-olds to examine how language abilities change with age. This kind of question would be best answered using a cross-sectional research design.

Longitudinal research designs

Longitudinal research designs are used to examine behavior in the same infants and children over time. For example, when considering our example of hide-and-seek behaviors in preschoolers, a researcher might conduct a longitudinal study to examine whether 2-year-olds develop into better hiders over time. To this end, a researcher might observe a group of 2-year-old children playing hide-and-seek with plans to observe them again when they are 4 years old – and again when they are 6 years old. This study is longitudinal in nature because the researcher plans to study the same children as they age. Based on her data, the researcher might conclude that 2-year-olds develop more mature hiding abilities with age. Remember, researchers examine games such as hide-and-seek not because they are interested in the games themselves, but because they offer clues to how children think, feel and behave at various ages.

Longitudinal studies may be conducted over the short term (over a span of months, as in Wiebe, Lukowski, & Bauer, 2010) or over much longer durations (years or decades, as in Lukowski et al., 2010). For these reasons, longitudinal research designs are optimal for studying stability and change over time. Longitudinal research also has limitations, however. For one, longitudinal studies are expensive: they require that researchers maintain continued contact with participants over time, and they necessitate that scientists have funding to conduct their work over extended durations (from infancy to when participants were 19 years old in Lukowski et al., 2010). An additional risk is attrition . Attrition occurs when participants fail to complete all portions of a study. Participants may move, change their phone numbers, or simply become disinterested in participating over time. Researchers should account for the possibility of attrition by enrolling a larger sample into their study initially, as some participants will likely drop out over time.

The results from longitudinal studies may also be impacted by repeated assessments. Consider how well you would do on a math test if you were given the exact same exam every day for a week. Your performance would likely improve over time not necessarily because you developed better math abilities, but because you were continuously practicing the same math problems. This phenomenon is known as a practice effect . Practice effects occur when participants become better at a task over time because they have done it again and again; not due to natural psychological development. A final limitation of longitudinal research is that the results may be impacted by cohort effects . Cohort effects occur when the results of the study are affected by the particular point in historical time during which participants are tested. As an example, think about how peer relationships in childhood have likely changed since February 2004 – the month and year Facebook was founded. Cohort effects can be problematic in longitudinal research because only one group of participants are tested at one point in time – different findings might be expected if participants of the same ages were tested at different points in historical time.

Cross-sectional designs

Cross-sectional research designs are used to examine behavior in participants of different ages who are tested at the same point in time. When considering our example of hide-and-seek behaviors in children, for example, a researcher might want to examine whether older children more often hide in novel locations (those in which another child in the same game has never hidden before) when compared to younger children. In this case, the researcher might observe 2-, 4-, and 6-year-old children as they play the game (the various age groups represent the “cross sections”). This research is cross-sectional in nature because the researcher plans to examine the behavior of children of different ages within the same study at the same time. Based on her data, the researcher might conclude that 2-year-olds more commonly hide in previously-searched locations relative to 6-year-olds.

Cross-sectional designs are useful for many reasons. Because participants of different ages are tested at the same point in time, data collection can proceed at a rapid pace. In addition, because participants are only tested at one point in time, practice effects are not an issue – children do not have the opportunity to become better at the task over time. Cross-sectional designs are also more cost-effective than longitudinal research designs because there is no need to maintain contact with and follow-up on participants over time.

One of the primary limitations of cross-sectional research, however, is that the results yield information on age-related change, not development per se . That is, although the study described above can show that 6-year-olds are more advanced in their hiding behavior than 2-year-olds, the data used to come up with this conclusion were collected from different children. It could be, for instance, that this specific sample of 6-year-olds just happened to be particularly clever at hide-and-seek. As such, the researcher cannot conclude that 2-year-olds develop into better hiders with age; she can only state that 6-year-olds, on average, are more sophisticated hiders relative to children 4 years younger.

Sequential research designs

Sequential research designs include elements of both longitudinal and cross-sectional research designs. Similar to longitudinal designs, sequential research features participants who are followed over time; similar to cross-sectional designs, sequential work includes participants of different ages. This research design is also distinct from those that have been discussed previously in that children of different ages are enrolled into a study at various points in time to examine age-related changes, development within the same individuals as they age, and account for the possibility of cohort effects.

Consider, once again, our example of hide-and-seek behaviors. In a study with a sequential design, a researcher might enroll three separate groups of children (Groups A, B, and C). Children in Group A would be enrolled when they are 2 years old and would be tested again when they are 4 and 6 years old (similar in design to the longitudinal study described previously). Children in Group B would be enrolled when they are 4 years old and would be tested again when they are 6 and 8 years old. Finally, children in Group C would be enrolled when they are 6 years old and would be tested again when they are 8 and 10 years old.

Studies with sequential designs are powerful because they allow for both longitudinal and cross-sectional comparisons. This research design also allows for the examination of cohort effects. For example, the researcher could examine the hide-and-seek behavior of 6-year-olds in Groups A, B, and C to determine whether performance differed by group when participants were the same age. If performance differences were found, there would be evidence for a cohort effect. In the hide-and-seek example, this might mean that children from different time periods varied in the amount they giggled or how patient they are when waiting to be found. Sequential designs are also appealing because they allow researchers to learn a lot about development in a relatively short amount of time. In the previous example, a four-year research study would provide information about 8 years of developmental time by enrolling children ranging in age from two to ten years old.

Because they include elements of longitudinal and cross-sectional designs, sequential research has many of the same strengths and limitations as these other approaches. For example, sequential work may require less time and effort than longitudinal research, but more time and effort than cross-sectional research. Although practice effects may be an issue if participants are asked to complete the same tasks or assessments over time, attrition may be less problematic than what is commonly experienced in longitudinal research since participants may not have to remain involved in the study for such a long period of time.

When considering the best research design to use in their research, scientists think about their main research question and the best way to come up with an answer. A table of advantages and disadvantages for each of the described research designs is provided here to help you as you consider what sorts of studies would be best conducted using each of these different approaches.

Challenges Associated with Conducting Developmental Research

The previous sections describe research tools to assess development in infancy and early childhood, as well as the ways that research designs can be used to track age-related changes and development over time. Before you begin conducting developmental research, however, you must also be aware that testing infants and children comes with its own unique set of challenges. In the final section of this module, we review some of the main issues that are encountered when conducting research with the youngest of human participants. In particular, we focus our discussion on ethical concerns, recruitment issues, and participant attrition.

Ethical concerns

As a student of psychological science, you may already know that Institutional Review Boards (IRBs) review and approve of all research projects that are conducted at universities, hospitals, and other institutions. An IRB is typically a panel of experts who read and evaluate proposals for research. IRB members want to ensure that the proposed research will be carried out ethically and that the potential benefits of the research outweigh the risks and harm for participants. What you may not know though, is that the IRB considers some groups of participants to be more vulnerable or at-risk than others. Whereas university students are generally not viewed as vulnerable or at-risk, infants and young children commonly fall into this category. What makes infants and young children more vulnerable during research than young adults? One reason infants and young children are perceived as being at increased risk is due to their limited cognitive capabilities, which makes them unable to state their willingness to participate in research or tell researchers when they would like to drop out of a study. For these reasons, infants and young children require special accommodations as they participate in the research process.

When thinking about special accommodations in developmental research, consider the informed consent process. If you have ever participated in psychological research, you may know through your own experience that adults commonly sign an informed consent statement (a contract stating that they agree to participate in research) after learning about a study. As part of this process, participants are informed of the procedures to be used in the research, along with any expected risks or benefits. Infants and young children cannot verbally indicate their willingness to participate, much less understand the balance of potential risks and benefits. As such, researchers are oftentimes required to obtain written informed consent from the parent or legal guardian of the child participant, an adult who is almost always present as the study is conducted. In fact, children are not asked to indicate whether they would like to be involved in a study at all (a process known as assent ) until they are approximately seven years old. Because infants and young children also cannot easily indicate if they would like to discontinue their participation in a study, researchers must be sensitive to changes in the state of the participant (determining whether a child is too tired or upset to continue) as well as to parent desires (in some cases, parents might want to discontinue their involvement in the research). As in adult studies, researchers must always strive to protect the rights and well-being of the minor participants and their parents when conducting developmental science.

Recruitment

An additional challenge in developmental science is participant recruitment. Recruiting university students to participate in adult studies is typically easy. Many colleges and universities offer extra credit for participation in research and have locations such as bulletin boards and school newspapers where research can be advertised. Unfortunately, young children cannot be recruited by making announcements in Introduction to Psychology courses, by posting ads on campuses, or through online platforms such as Amazon Mechanical Turk. Given these limitations, how do researchers go about finding infants and young children to be in their studies?

The answer to this question varies along multiple dimensions. Researchers must consider the number of participants they need and the financial resources available to them, among other things. Location may also be an important consideration. Researchers who need large numbers of infants and children may attempt to do so by obtaining infant birth records from the state, county, or province in which they reside. Some areas make this information publicly available for free, whereas birth records must be purchased in other areas (and in some locations birth records may be entirely unavailable as a recruitment tool). If birth records are available, researchers can use the obtained information to call families by phone or mail them letters describing possible research opportunities. All is not lost if this recruitment strategy is unavailable, however. Researchers can choose to pay a recruitment agency to contact and recruit families for them. Although these methods tend to be quick and effective, they can also be quite expensive. More economical recruitment options include posting advertisements and fliers in locations frequented by families, such as mommy-and-me classes, local malls, and preschools or day care centers. Researchers can also utilize online social media outlets like Facebook, which allows users to post recruitment advertisements for a small fee. Of course, each of these different recruitment techniques requires IRB approval.

Another important consideration when conducting research with infants and young children is attrition . Although attrition is quite common in longitudinal research in particular, it is also problematic in developmental science more generally, as studies with infants and young children tend to have higher attrition rates than studies with adults. For example, high attrition rates in ERP studies oftentimes result from the demands of the task: infants are required to sit still and have a tight, wet cap placed on their heads before watching still photographs on a computer screen in a dark, quiet room. In other cases, attrition may be due to motivation (or a lack thereof). Whereas adults may be motivated to participate in research in order to receive money or extra course credit, infants and young children are not as easily enticed. In addition, infants and young children are more likely to tire easily, become fussy, and lose interest in the study procedures than are adults. For these reasons, research studies should be designed to be as short as possible – it is likely better to break up a large study into multiple short sessions rather than cram all of the tasks into one long visit to the lab. Researchers should also allow time for breaks in their study protocols so that infants can rest or have snacks as needed. Happy, comfortable participants provide the best data.

Conclusions

Child development is a fascinating field of study – but care must be taken to ensure that researchers use appropriate methods to examine infant and child behavior, use the correct experimental design to answer their questions, and be aware of the special challenges that are part-and-parcel of developmental research. After reading this module, you should have a solid understanding of these various issues and be ready to think more critically about research questions that interest you. For example, when considering our initial example of hide-and-seek behaviors in preschoolers, you might ask questions about what other factors might contribute to hiding behaviors in children. Do children with older siblings hide in locations that were previously searched less often than children without siblings? What other abilities are associated with the development of hiding skills? Do children who use more sophisticated hiding strategies as preschoolers do better on other tests of cognitive functioning in high school? Many interesting questions remain to be examined by future generations of developmental scientists – maybe you will make one of the next big discoveries!

Outside Resources

Discussion Questions

• Why is it important to conduct research on infants and children?
• What are some possible benefits and limitations of the various research methods discussed in this module?
• Why is it important to examine cohort effects in developmental research?
• Think about additional challenges or unique issues that might be experienced by developmental scientists. How would they handle the challenges and issues you’ve addressed?
• Work with your peers to design a study to identify whether children who were good hiders as preschoolers are more cognitively advanced in high school. What research design would you use and why? What are the advantages and limitations of the design you selected?
• Achenbach, T. M., & Rescorla, L. A. (2000). Manual for the ASEBA preschool forms and profiles: An integrated system of multi-informant assessment. Burlington, VT: University of Vermont Department of Psychiatry.
• Baillargeon, R. (1987). Object permanence in 3½- and 4½-month-old infants. Developmental Psychology, 23, 655-664. doi: 10.1037/0012-1649.23.5.655
• Baillargeon, R., Spelke, E., & Wasserman, S. (1985). Object permanence in five-month-old infants. Cognition, 20, 191-208. doi: 10.1016/0010-0277(85)90008-3
• Barr, R., Dowden, A., & Hayne, H. (1996). Developmental changes in deferred imitation by 6- to 24-month-old infants. Infant Behavior and Development, 19 , 159-170. doi: 10.1016/s0163-6383(96)90015-6
• Bauer, P. J., Wenner, J. A., Dropik, P. L., & Wewerka, S. S. (2000). Parameters of remembering and forgetting in the transition from infancy to early childhood. Monographs of the Society for Research in Child Development, 65 , 1-204. doi: 10.1016/j.imlet.2014.04.001
• Bauer, P. J., Wiebe, S. A., Carver, L. J., Waters, J. M., & Nelson, C. A. (2003). Developments in long-term explicit memory late in the first year of life: Behavioral and electrophysiological indices. Psychological Science, 14 , 629-635. doi: 10.1046/j.0956-7976.2003.psci_1476.x
• Bauer, P. J., Wiebe, S. A., Waters, J. M., & Bangston, S. K. (2001). Reexposure breeds recall: Effects of experience on 9-month-olds’ ordered recall. Journal of Experimental Child Psychology, 80 , 174-200. doi: 10.1006/jecp.2000.2628
• Bentin, S., Allison, T., Puce, A., Perez, E., & McCarthy, G. (2010). Electrophysiological studies of face perception in humans. Journal of Cognitive Neuroscience , 8, 551-565. doi: 10.1162/jocn.1996.8.6.551
• Carver, L. J., & Bauer, P. J. (1999). When the event is more than the sum of its parts: 9-month-olds’ long-term ordered recall. Memory, 7 , 147-174. doi: 10.1080/741944070
• Cashon, C. H., & Cohen, L. B. (2000). Eight-month-old infants’ perception of possible and impossible events. Infancy, 1 , 429-446. doi: 10.1016/s0163-6383(98)91561-2
• Collie, R., & Hayne, H. (1999). Deferred imitation by 6- and 9-month-old infants: More evidence for declarative memory. Developmental Psychobiology, 35 , 83-90. doi: 10.1002/(sici)1098-2302(199909)35:2<83::aid-dev1>3.0.co;2-s
• DeBoer, T., Scott, L. S., & Nelson, C. A. (2005). ERPs in developmental populations. In T. C. Handy (Ed.), Event-related potentials: A methods handbook (pp. 263-297) . Cambridge, MA: The MIT Press.
• Lukowski, A. F., & Milojevich, H. M. (2016). Examining recall memory in infancy and early childhood using the elicited imitation paradigm. Journal of Visualized Experiments, 110 , e53347.
• Lukowski, A. F., Koss, M., Burden, M. J., Jonides, J., Nelson, C. A., Kaciroti, N., … Lozoff, B. (2010). Iron deficiency in infancy and neurocognitive functioning at 19 years: Evidence of long-term deficits in executive function and recognition memory. Nutritional Neuroscience, 13 , 54-70. doi: 10.1179/147683010x12611460763689
• Ronald, A., Pennell, C. E., & Whitehouse, A. J. O. (2011). Prenatal maternal stress associated with ADHD and autistic traits in early childhood. Frontiers in Psychology, 1 , 1-8. doi: 10.3389/fpsyg.2010.00223
• Walrath, R. (2011). Kohlberg’s theory of moral development. In Encyclopedia of Child Behavior and Development (pp. 859–860).
• Webb, S. J., Dawson, G., Bernier, R., & Panagiotides, H. (2006). ERP evidence of atypical face processing in young children with autism. Journal of Autism and Developmental Disorders, 36 , 884-890. doi: 10.1007/s10803-006-0126-x
• Wiebe, S. A., Lukowski, A. F., & Bauer, P. J. (2010). Sequence imitation and reaching measures of executive control: A longitudinal examination in the second year of life. Developmental Neuropsychology, 35 , 522-538. doi: 10.1080/87565641.2010.494751

42. Developmental Research

The field of instructional technology has traditionally involved a unique blend of theory and practice. This blend is most obvious in developmental research, which involves the production of knowledge with the ultimate aim of improving the processes of instructional design, development, and evaluation. It is based on either situation-specific problem solving or generalized inquiry procedures. Developmental research, as opposed to simple instructional development, has been defined as "the systematic study of designing, developing and evaluating instructional programs, processes, and products that must meet the criteria of internal consistency and effectiveness" (Seels & Richey, 1994, p. 127). In its simplest form, developmental research could be either:

• A situation in which someone is performing instructional design, development, or evaluation activities and studying the process at the same time
• The study of the impact of someone else's instructional design and development efforts
• The study of the instructional design, development, and evaluation process as a whole, or of particular process components

In each case, the distinction is made between performing a process and studying that process. Reports of developmental research may take the form of a case study with retrospective analysis, an evaluation report, or even that of a typical experimental research report.

The purposes of this chapter(see footnote ) are to:

• Explore the nature and background of developmental research
• Describe the major types of developmental research by examining a range of representative projects
• Analyze the methodological approaches used in the various types of developmental research
• Describe the issues, findings, and trends in recent developmental research
• Discuss the future of this type of research in our field

Research in Developmental Psychology

What you’ll learn to do: examine how to do research in lifespan development.

How do we know what changes and stays the same (and when and why) in lifespan development? We rely on research that utilizes the scientific method so that we can have confidence in the findings. How data are collected may vary by age group and by the type of information sought. The developmental design (for example, following individuals as they age over time or comparing individuals of different ages at one point in time) will affect the data and the conclusions that can be drawn from them about actual age changes. What do you think are the particular challenges or issues in conducting developmental research, such as with infants and children? Read on to learn more.

Learning outcomes

• Explain how the scientific method is used in researching development
• Compare various types and objectives of developmental research
• Describe methods for collecting research data (including observation, survey, case study, content analysis, and secondary content analysis)
• Explain correlational research
• Describe the value of experimental research
• Compare the advantages and disadvantages of developmental research designs (cross-sectional, longitudinal, and sequential)
• Describe challenges associated with conducting research in lifespan development

Research in Lifespan Development

How do we know what we know.

An important part of learning any science is having a basic knowledge of the techniques used in gathering information. The hallmark of scientific investigation is that of following a set of procedures designed to keep questioning or skepticism alive while describing, explaining, or testing any phenomenon. Not long ago a friend said to me that he did not trust academicians or researchers because they always seem to change their story. That, however, is exactly what science is all about; it involves continuously renewing our understanding of the subjects in question and an ongoing investigation of how and why events occur. Science is a vehicle for going on a never-ending journey. In the area of development, we have seen changes in recommendations for nutrition, in explanations of psychological states as people age, and in parenting advice. So think of learning about human development as a lifelong endeavor.

Personal Knowledge

How do we know what we know? Take a moment to write down two things that you know about childhood. Okay. Now, how do you know? Chances are you know these things based on your own history (experiential reality), what others have told you, or cultural ideas (agreement reality) (Seccombe and Warner, 2004). There are several problems with personal inquiry or drawing conclusions based on our personal experiences.

Our assumptions very often guide our perceptions, consequently, when we believe something, we tend to see it even if it is not there. Have you heard the saying, “seeing is believing”? Well, the truth is just the opposite: believing is seeing. This problem may just be a result of cognitive ‘blinders’ or it may be part of a more conscious attempt to support our own views. Confirmation bias is the tendency to look for evidence that we are right and in so doing, we ignore contradictory evidence.

Philosopher Karl Popper suggested that the distinction between that which is scientific and that which is unscientific is that science is falsifiable; scientific inquiry involves attempts to reject or refute a theory or set of assumptions (Thornton, 2005). A theory that cannot be falsified is not scientific. And much of what we do in personal inquiry involves drawing conclusions based on what we have personally experienced or validating our own experience by discussing what we think is true with others who share the same views.

Science offers a more systematic way to make comparisons and guard against bias. One technique used to avoid sampling bias is to select participants for a study in a random way. This means using a technique to ensure that all members have an equal chance of being selected. Simple random sampling may involve using a set of random numbers as a guide in determining who is to be selected. For example, if we have a list of 400 people and wish to randomly select a smaller group or sample to be studied, we use a list of random numbers and select the case that corresponds with that number (Case 39, 3, 217, etc.). This is preferable to asking only those individuals with whom we are familiar to participate in a study; if we conveniently chose only people we know, we know nothing about those who had no opportunity to be selected. There are many more elaborate techniques that can be used to obtain samples that represent the composition of the population we are studying. But even though a randomly selected representative sample is preferable, it is not always used because of costs and other limitations. As a consumer of research, however, you should know how the sample was obtained and keep this in mind when interpreting results. It is possible that what was found was limited to that sample or similar individuals and not generalizable to everyone else.

Scientific Methods

The particular method used to conduct research may vary by discipline and since lifespan development is multidisciplinary, more than one method may be used to study human development. One method of scientific investigation involves the following steps:

• Determining a research question
• Reviewing previous studies addressing the topic in question (known as a literature review)
• Determining a method of gathering information
• Conducting the study
• Interpreting the results
• Drawing conclusions; stating limitations of the study and suggestions for future research
• Making the findings available to others (both to share information and to have the work scrutinized by others)

The findings of these scientific studies can then be used by others as they explore the area of interest. Through this process, a literature or knowledge base is established. This model of scientific investigation presents research as a linear process guided by a specific research question. And it typically involves quantitative research , which relies on numerical data or using statistics to understand and report what has been studied.

Another model of research, referred to as qualitative research, may involve steps such as these:

• Begin with a broad area of interest and a research question
• Gain entrance into a group to be researched
• Gather field notes about the setting, the people, the structure, the activities, or other areas of interest
• Ask open-ended, broad “grand tour” types of questions when interviewing subjects
• Modify research questions as the study continues
• Note patterns or consistencies
• Explore new areas deemed important by the people being observed
• Report findings

In this type of research, theoretical ideas are “grounded” in the experiences of the participants. The researcher is the student and the people in the setting are the teachers as they inform the researcher of their world (Glazer & Strauss, 1967). Researchers should be aware of their own biases and assumptions, acknowledge them, and bracket them in efforts to keep them from limiting accuracy in reporting. Sometimes qualitative studies are used initially to explore a topic and more quantitative studies are used to test or explain what was first described.

A good way to become more familiar with these scientific research methods, both quantitative and qualitative, is to look at journal articles, which are written in sections that follow these steps in the scientific process. Most psychological articles and many papers in the social sciences follow the writing guidelines and format dictated by the  American Psychological Association  (APA). In general, the structure follows: abstract (summary of the article), introduction or literature review, methods explaining how the study was conducted, results of the study, discussion and interpretation of findings, and references.

Link to Learning

Brené Brown is a bestselling author and social work professor at the University of Houston. She conducts grounded theory research by collecting qualitative data from large numbers of participants. In Brené Brown’s TED Talk The Power of Vulnerability , Brown refers to herself as a storyteller-researcher as she explains her research process and summarizes her results.

Research Methods and Objectives

The main categories of psychological research are descriptive, correlational, and experimental research. Research studies that do not test specific relationships between variables are called  descriptive, or qualitative, studies . These studies are used to describe general or specific behaviors and attributes that are observed and measured. In the early stages of research, it might be difficult to form a hypothesis, especially when there is not any existing literature in the area. In these situations designing an experiment would be premature, as the question of interest is not yet clearly defined as a hypothesis. Often a researcher will begin with a non-experimental approach, such as a descriptive study, to gather more information about the topic before designing an experiment or correlational study to address a specific hypothesis. Some examples of descriptive questions include:

• “How much time do parents spend with their children?”
• “How many times per week do couples have intercourse?”
• “When is marital satisfaction greatest?”

The main types of descriptive studies include observation, case studies, surveys, and content analysis (which we’ll examine further in the module). Descriptive research is distinct from  correlational research , in which psychologists formally test whether a relationship exists between two or more variables.  Experimental research  goes a step further beyond descriptive and correlational research and randomly assigns people to different conditions, using hypothesis testing to make inferences about how these conditions affect behavior. Some experimental research includes explanatory studies, which are efforts to answer the question “why” such as:

• “Why have rates of divorce leveled off?”
• “Why are teen pregnancy rates down?”
• “Why has the average life expectancy increased?”

Evaluation research is designed to assess the effectiveness of policies or programs. For instance, research might be designed to study the effectiveness of safety programs implemented in schools for installing car seats or fitting bicycle helmets. Do children who have been exposed to the safety programs wear their helmets? Do parents use car seats properly? If not, why not?

Research Methods

We have just learned about some of the various models and objectives of research in lifespan development. Now we’ll dig deeper to understand the methods and techniques used to describe, explain, or evaluate behavior.

All types of research methods have unique strengths and weaknesses, and each method may only be appropriate for certain types of research questions. For example, studies that rely primarily on observation produce incredible amounts of information, but the ability to apply this information to the larger population is somewhat limited because of small sample sizes. Survey research, on the other hand, allows researchers to easily collect data from relatively large samples. While this allows for results to be generalized to the larger population more easily, the information that can be collected on any given survey is somewhat limited and subject to problems associated with any type of self-reported data. Some researchers conduct archival research by using existing records. While this can be a fairly inexpensive way to collect data that can provide insight into a number of research questions, researchers using this approach have no control over how or what kind of data was collected.

Types of Descriptive Research

Observation.

Observational studies , also called naturalistic observation, involve watching and recording the actions of participants. This may take place in the natural setting, such as observing children at play in a park, or behind a one-way glass while children are at play in a laboratory playroom. The researcher may follow a checklist and record the frequency and duration of events (perhaps how many conflicts occur among 2-year-olds) or may observe and record as much as possible about an event as a participant (such as attending an Alcoholics Anonymous meeting and recording the slogans on the walls, the structure of the meeting, the expressions commonly used, etc.). The researcher may be a participant or a non-participant. What would be the strengths of being a participant? What would be the weaknesses?

In general, observational studies have the strength of allowing the researcher to see how people behave rather than relying on self-report. One weakness of self-report studies is that what people do and what they say they do are often very different. A major weakness of observational studies is that they do not allow the researcher to explain causal relationships. Yet, observational studies are useful and widely used when studying children. It is important to remember that most people tend to change their behavior when they know they are being watched (known as the Hawthorne effect ) and children may not survey well.

Case Studies

Case studies  involve exploring a single case or situation in great detail. Information may be gathered with the use of observation, interviews, testing, or other methods to uncover as much as possible about a person or situation. Case studies are helpful when investigating unusual situations such as brain trauma or children reared in isolation. And they are often used by clinicians who conduct case studies as part of their normal practice when gathering information about a client or patient coming in for treatment. Case studies can be used to explore areas about which little is known and can provide rich detail about situations or conditions. However, the findings from case studies cannot be generalized or applied to larger populations; this is because cases are not randomly selected and no control group is used for comparison. (Read The Man Who Mistook His Wife for a Hat by Dr. Oliver Sacks as a good example of the case study approach.)

Surveys  are familiar to most people because they are so widely used. Surveys enhance accessibility to subjects because they can be conducted in person, over the phone, through the mail, or online. A survey involves asking a standard set of questions to a group of subjects. In a highly structured survey, subjects are forced to choose from a response set such as “strongly disagree, disagree, undecided, agree, strongly agree”; or “0, 1-5, 6-10, etc.” Surveys are commonly used by sociologists, marketing researchers, political scientists, therapists, and others to gather information on many variables in a relatively short period of time. Surveys typically yield surface information on a wide variety of factors, but may not allow for an in-depth understanding of human behavior.

Surveys are useful in examining stated values, attitudes, opinions, and reporting on practices. However, they are based on self-report, or what people say they do rather than on observation, and this can limit accuracy. Validity refers to accuracy and reliability refers to consistency in responses to tests and other measures; great care is taken to ensure the validity and reliability of surveys.

Content Analysis

Content analysis  involves looking at media such as old texts, pictures, commercials, lyrics, or other materials to explore patterns or themes in culture. An example of content analysis is the classic history of childhood by Aries (1962) called “Centuries of Childhood” or the analysis of television commercials for sexual or violent content or for ageism. Passages in text or television programs can be randomly selected for analysis as well. Again, one advantage of analyzing work such as this is that the researcher does not have to go through the time and expense of finding respondents, but the researcher cannot know how accurately the media reflects the actions and sentiments of the population.

Secondary content analysis, or archival research, involves analyzing information that has already been collected or examining documents or media to uncover attitudes, practices, or preferences. There are a number of data sets available to those who wish to conduct this type of research. The researcher conducting secondary analysis does not have to recruit subjects but does need to know the quality of the information collected in the original study. And unfortunately, the researcher is limited to the questions asked and data collected originally.

Correlational and Experimental Research

Correlational research.

When scientists passively observe and measure phenomena it is called correlational research . Here, researchers do not intervene and change behavior, as they do in experiments. In correlational research, the goal is to identify patterns of relationships, but not cause and effect. Importantly, with correlational research, you can examine only two variables at a time, no more and no less.

So, what if you wanted to test whether spending money on others is related to happiness, but you don’t have $20 to give to each participant in order to have them spend it for your experiment? You could use a correlational design—which is exactly what Professor Elizabeth Dunn (2008) at the University of British Columbia did when she conducted research on spending and happiness. She asked people how much of their income they spent on others or donated to charity, and later she asked them how happy they were. Do you think these two variables were related? Yes, they were! The more money people reported spending on others, the happier they were.

Understanding Correlation

With a positive correlation , the two variables go up or down together. In a scatterplot, the dots form a pattern that extends from the bottom left to the upper right (just as they do in Figure 1). The r value for a positive correlation is indicated by a positive number (although, the positive sign is usually omitted). Here, the r value is .81. For the example above, the direction of the association is positive. This means that people who perceived the past month as being good reported feeling happier, whereas people who perceived the month as being bad reported feeling less happy.

A negative correlation is one in which the two variables move in opposite directions. That is, as one variable goes up, the other goes down. Figure 2 shows the association between the average height of males in a country (y-axis) and the pathogen prevalence (or commonness of disease; x-axis) of that country. In this scatterplot, each dot represents a country. Notice how the dots extend from the top left to the bottom right. What does this mean in real-world terms? It means that people are shorter in parts of the world where there is more disease. The r-value for a negative correlation is indicated by a negative number—that is, it has a minus (–) sign in front of it. Here, it is –.83.

Experimental Research

Experiments  are designed to test  hypotheses  (or specific statements about the relationship between  variables ) in a controlled setting in an effort to explain how certain factors or events produce outcomes. A variable is anything that changes in value. Concepts are operationalized  or transformed into variables in research which means that the researcher must specify exactly what is going to be measured in the study. For example, if we are interested in studying marital satisfaction, we have to specify what marital satisfaction really means or what we are going to use as an indicator of marital satisfaction. What is something measurable that would indicate some level of marital satisfaction? Would it be the amount of time couples spend together each day? Or eye contact during a discussion about money? Or maybe a subject’s score on a marital satisfaction scale? Each of these is measurable but these may not be equally valid or accurate indicators of marital satisfaction. What do you think? These are the kinds of considerations researchers must make when working through the design.

The experimental method is the only research method that can measure cause and effect relationships between variables. Three conditions must be met in order to establish cause and effect. Experimental designs are useful in meeting these conditions:

• The independent and dependent variables must be related.  In other words, when one is altered, the other changes in response. The independent variable is something altered or introduced by the researcher; sometimes thought of as the treatment or intervention. The dependent variable is the outcome or the factor affected by the introduction of the independent variable; the dependent variable  depends on the independent variable. For example, if we are looking at the impact of exercise on stress levels, the independent variable would be exercise; the dependent variable would be stress.
• The cause must come before the effect.  Experiments measure subjects on the dependent variable before exposing them to the independent variable (establishing a baseline). So we would measure the subjects’ level of stress before introducing exercise and then again after the exercise to see if there has been a change in stress levels. (Observational and survey research does not always allow us to look at the timing of these events which makes understanding causality problematic with these methods.)
• The cause must be isolated.  The researcher must ensure that no outside, perhaps unknown variables, are actually causing the effect we see. The experimental design helps make this possible. In an experiment, we would make sure that our subjects’ diets were held constant throughout the exercise program. Otherwise, the diet might really be creating a change in stress level rather than exercise.

A basic experimental design involves beginning with a sample (or subset of a population) and randomly assigning subjects to one of two groups: the  experimental group or the control group . Ideally, to prevent bias, the participants would be blind to their condition (not aware of which group they are in) and the researchers would also be blind to each participant’s condition (referred to as “ double blind “). The experimental group is the group that is going to be exposed to an independent variable or condition the researcher is introducing as a potential cause of an event. The control group is going to be used for comparison and is going to have the same experience as the experimental group but will not be exposed to the independent variable. This helps address the placebo effect, which is that a group may expect changes to happen just by participating. After exposing the experimental group to the independent variable, the two groups are measured again to see if a change has occurred. If so, we are in a better position to suggest that the independent variable caused the change in the dependent variable . The basic experimental model looks like this:

The major advantage of the experimental design is that of helping to establish cause and effect relationships. A disadvantage of this design is the difficulty of translating much of what concerns us about human behavior into a laboratory setting.

Developmental Research Designs

Now you know about some tools used to conduct research about human development. Remember,  research methods  are tools that are used to collect information. But it is easy to confuse research methods and research design. Research design is the strategy or blueprint for deciding how to collect and analyze information. Research design dictates which methods are used and how. Developmental research designs are techniques used particularly in lifespan development research. When we are trying to describe development and change, the research designs become especially important because we are interested in what changes and what stays the same with age. These techniques try to examine how age, cohort, gender, and social class impact development.

Cross-sectional designs

The majority of developmental studies use cross-sectional designs because they are less time-consuming and less expensive than other developmental designs. Cross-sectional research designs are used to examine behavior in participants of different ages who are tested at the same point in time. Let’s suppose that researchers are interested in the relationship between intelligence and aging. They might have a hypothesis (an educated guess, based on theory or observations) that intelligence declines as people get older. The researchers might choose to give a certain intelligence test to individuals who are 20 years old, individuals who are 50 years old, and individuals who are 80 years old at the same time and compare the data from each age group. This research is cross-sectional in design because the researchers plan to examine the intelligence scores of individuals of different ages within the same study at the same time; they are taking a “cross-section” of people at one point in time. Let’s say that the comparisons find that the 80-year-old adults score lower on the intelligence test than the 50-year-old adults, and the 50-year-old adults score lower on the intelligence test than the 20-year-old adults. Based on these data, the researchers might conclude that individuals become less intelligent as they get older. Would that be a valid (accurate) interpretation of the results?

No, that would not be a valid conclusion because the researchers did not follow individuals as they aged from 20 to 50 to 80 years old. One of the primary limitations of cross-sectional research is that the results yield information about age differences  not necessarily changes with age or over time. That is, although the study described above can show that in 2010, the 80-year-olds scored lower on the intelligence test than the 50-year-olds, and the 50-year-olds scored lower on the intelligence test than the 20-year-olds, the data used to come up with this conclusion were collected from different individuals (or groups of individuals). It could be, for instance, that when these 20-year-olds get older (50 and eventually 80), they will still score just as high on the intelligence test as they did at age 20. In a similar way, maybe the 80-year-olds would have scored relatively low on the intelligence test even at ages 50 and 20; the researchers don’t know for certain because they did not follow the same individuals as they got older.

It is also possible that the differences found between the age groups are not due to age, per se, but due to cohort effects. The 80-year-olds in this 2010 research grew up during a particular time and experienced certain events as a group. They were born in 1930 and are part of the Traditional or Silent Generation. The 50-year-olds were born in 1960 and are members of the Baby Boomer cohort. The 20-year-olds were born in 1990 and are part of the Millennial or Gen Y Generation. What kinds of things did each of these cohorts experience that the others did not experience or at least not in the same ways?

You may have come up with many differences between these cohorts’ experiences, such as living through certain wars, political and social movements, economic conditions, advances in technology, changes in health and nutrition standards, etc. There may be particular cohort differences that could especially influence their performance on intelligence tests, such as education level and use of computers. That is, many of those born in 1930 probably did not complete high school; those born in 1960 may have high school degrees, on average, but the majority did not attain college degrees; the young adults are probably current college students. And this is not even considering additional factors such as gender, race, or socioeconomic status. The young adults are used to taking tests on computers, but the members of the other two cohorts did not grow up with computers and may not be as comfortable if the intelligence test is administered on computers. These factors could have been a factor in the research results.

Another disadvantage of cross-sectional research is that it is limited to one time of measurement. Data are collected at one point in time and it’s possible that something could have happened in that year in history that affected all of the participants, although possibly each cohort may have been affected differently. Just think about the mindsets of participants in research that was conducted in the United States right after the terrorist attacks on September 11, 2001.

Longitudinal research designs

Longitudinal   research involves beginning with a group of people who may be of the same age and background (cohort) and measuring them repeatedly over a long period of time. One of the benefits of this type of research is that people can be followed through time and be compared with themselves when they were younger; therefore changes with age over time are measured. What would be the advantages and disadvantages of longitudinal research? Problems with this type of research include being expensive, taking a long time, and subjects dropping out over time. Think about the film, 63 Up , part of the Up Series mentioned earlier, which is an example of following individuals over time. In the videos, filmed every seven years, you see how people change physically, emotionally, and socially through time; and some remain the same in certain ways, too. But many of the participants really disliked being part of the project and repeatedly threatened to quit; one disappeared for several years; another died before her 63rd year. Would you want to be interviewed every seven years? Would you want to have it made public for all to watch?   

Longitudinal research designs are used to examine behavior in the same individuals over time. For instance, with our example of studying intelligence and aging, a researcher might conduct a longitudinal study to examine whether 20-year-olds become less intelligent with age over time. To this end, a researcher might give an intelligence test to individuals when they are 20 years old, again when they are 50 years old, and then again when they are 80 years old. This study is longitudinal in nature because the researcher plans to study the same individuals as they age. Based on these data, the pattern of intelligence and age might look different than from the cross-sectional research; it might be found that participants’ intelligence scores are higher at age 50 than at age 20 and then remain stable or decline a little by age 80. How can that be when cross-sectional research revealed declines in intelligence with age?

Since longitudinal research happens over a period of time (which could be short term, as in months, but is often longer, as in years), there is a risk of attrition. Attrition occurs when participants fail to complete all portions of a study. Participants may move, change their phone numbers, die, or simply become disinterested in participating over time. Researchers should account for the possibility of attrition by enrolling a larger sample into their study initially, as some participants will likely drop out over time. There is also something known as  selective attrition— this means that certain groups of individuals may tend to drop out. It is often the least healthy, least educated, and lower socioeconomic participants who tend to drop out over time. That means that the remaining participants may no longer be representative of the whole population, as they are, in general, healthier, better educated, and have more money. This could be a factor in why our hypothetical research found a more optimistic picture of intelligence and aging as the years went by. What can researchers do about selective attrition? At each time of testing, they could randomly recruit more participants from the same cohort as the original members, to replace those who have dropped out.

The results from longitudinal studies may also be impacted by repeated assessments. Consider how well you would do on a math test if you were given the exact same exam every day for a week. Your performance would likely improve over time, not necessarily because you developed better math abilities, but because you were continuously practicing the same math problems. This phenomenon is known as a practice effect. Practice effects occur when participants become better at a task over time because they have done it again and again (not due to natural psychological development). So our participants may have become familiar with the intelligence test each time (and with the computerized testing administration). Another limitation of longitudinal research is that the data are limited to only one cohort.

Sequential research designs

Sequential research designs include elements of both longitudinal and cross-sectional research designs. Similar to longitudinal designs, sequential research features participants who are followed over time; similar to cross-sectional designs, sequential research includes participants of different ages. This research design is also distinct from those that have been discussed previously in that individuals of different ages are enrolled into a study at various points in time to examine age-related changes, development within the same individuals as they age, and to account for the possibility of cohort and/or time of measurement effects. In 1965, K. Warner Schaie described particular sequential designs: cross-sequential, cohort sequential, and time-sequential. The differences between them depended on which variables were focused on for analyses of the data (data could be viewed in terms of multiple cross-sectional designs or multiple longitudinal designs or multiple cohort designs). Ideally, by comparing results from the different types of analyses, the effects of age, cohort, and time in history could be separated out.

Challenges Conducting Developmental Research

The previous sections describe research tools to assess development across the lifespan, as well as the ways that research designs can be used to track age-related changes and development over time. Before you begin conducting developmental research, however, you must also be aware that testing individuals of certain ages (such as infants and children) or making comparisons across ages (such as children compared to teens) comes with its own unique set of challenges. In the final section of this module, let’s look at some of the main issues that are encountered when conducting developmental research, namely ethical concerns, recruitment issues, and participant attrition.

Ethical Concerns

You may already know that Institutional Review Boards (IRBs) must review and approve all research projects that are conducted at universities, hospitals, and other institutions (each broad discipline or field, such as psychology or social work, often has its own code of ethics that must also be followed, regardless of institutional affiliation). An IRB is typically a panel of experts who read and evaluate proposals for research. IRB members want to ensure that the proposed research will be carried out ethically and that the potential benefits of the research outweigh the risks and potential harm (psychological as well as physical harm) for participants.

What you may not know though, is that the IRB considers some groups of participants to be more vulnerable or at-risk than others. Whereas university students are generally not viewed as vulnerable or at-risk, infants and young children commonly fall into this category. What makes infants and young children more vulnerable during research than young adults? One reason infants and young children are perceived as being at increased risk is due to their limited cognitive capabilities, which makes them unable to state their willingness to participate in research or tell researchers when they would like to drop out of a study. For these reasons, infants and young children require special accommodations as they participate in the research process. Similar issues and accommodations would apply to adults who are deemed to be of limited cognitive capabilities.

When thinking about special accommodations in developmental research, consider the informed consent process. If you have ever participated in scientific research, you may know through your own experience that adults commonly sign an informed consent statement (a contract stating that they agree to participate in research) after learning about a study. As part of this process, participants are informed of the procedures to be used in the research, along with any expected risks or benefits. Infants and young children cannot verbally indicate their willingness to participate, much less understand the balance of potential risks and benefits. As such, researchers are oftentimes required to obtain written informed consent from the parent or legal guardian of the child participant, an adult who is almost always present as the study is conducted. In fact, children are not asked to indicate whether they would like to be involved in a study at all (a process known as assent) until they are approximately seven years old. Because infants and young children cannot easily indicate if they would like to discontinue their participation in a study, researchers must be sensitive to changes in the state of the participant (determining whether a child is too tired or upset to continue) as well as to parent desires (in some cases, parents might want to discontinue their involvement in the research). As in adult studies, researchers must always strive to protect the rights and well-being of the minor participants and their parents when conducting developmental research.

Recruitment

An additional challenge in developmental science is participant recruitment. Recruiting university students to participate in adult studies is typically easy.  Unfortunately, young children cannot be recruited in this way. Given these limitations, how do researchers go about finding infants and young children to be in their studies?

The answer to this question varies along multiple dimensions. Researchers must consider the number of participants they need and the financial resources available to them, among other things. Location may also be an important consideration. Researchers who need large numbers of infants and children may attempt to recruit them by obtaining infant birth records from the state, county, or province in which they reside. Researchers can choose to pay a recruitment agency to contact and recruit families for them.  More economical recruitment options include posting advertisements and fliers in locations frequented by families, such as mommy-and-me classes, local malls, and preschools or daycare centers. Researchers can also utilize online social media outlets like Facebook, which allows users to post recruitment advertisements for a small fee. Of course, each of these different recruitment techniques requires IRB approval. And if children are recruited and/or tested in school settings, permission would need to be obtained ahead of time from teachers, schools, and school districts (as well as informed consent from parents or guardians).

And what about the recruitment of adults? While it is easy to recruit young college students to participate in research, some would argue that it is too easy and that college students are samples of convenience. They are not randomly selected from the wider population, and they may not represent all young adults in our society (this was particularly true in the past with certain cohorts, as college students tended to be mainly white males of high socioeconomic status). In fact, in the early research on aging, this type of convenience sample was compared with another type of convenience sample—young college students tended to be compared with residents of nursing homes! Fortunately, it didn’t take long for researchers to realize that older adults in nursing homes are not representative of the older population; they tend to be the oldest and sickest (physically and/or psychologically). Those initial studies probably painted an overly negative view of aging, as young adults in college were being compared to older adults who were not healthy, had not been in school nor taken tests in many decades, and probably did not graduate high school, let alone college. As we can see, recruitment and random sampling can be significant issues in research with adults, as well as infants and children. For instance, how and where would you recruit middle-aged adults to participate in your research?

Another important consideration when conducting research with infants and young children is attrition . Although attrition is quite common in longitudinal research in particular (see the previous section on longitudinal designs for an example of high attrition rates and selective attrition in lifespan developmental research), it is also problematic in developmental science more generally, as studies with infants and young children tend to have higher attrition rates than studies with adults.  Infants and young children are more likely to tire easily, become fussy, and lose interest in the study procedures than are adults. For these reasons, research studies should be designed to be as short as possible – it is likely better to break up a large study into multiple short sessions rather than cram all of the tasks into one long visit to the lab. Researchers should also allow time for breaks in their study protocols so that infants can rest or have snacks as needed. Happy, comfortable participants provide the best data.

Conclusions

Lifespan development is a fascinating field of study – but care must be taken to ensure that researchers use appropriate methods to examine human behavior, use the correct experimental design to answer their questions, and be aware of the special challenges that are part-and-parcel of developmental research. After reading this module, you should have a solid understanding of these various issues and be ready to think more critically about research questions that interest you. For example, what types of questions do you have about lifespan development? What types of research would you like to conduct? Many interesting questions remain to be examined by future generations of developmental scientists – maybe you will make one of the next big discoveries!

Lifespan development is the scientific study of how and why people change or remain the same over time. As we are beginning to see, lifespan development involves multiple domains and many ages and stages that are important in and of themselves, but that are also interdependent and dynamic and need to be viewed holistically. There are many influences on lifespan development at individual and societal levels (including genetics); cultural, generational, economic, and historical contexts are often significant. And how developmental research is designed and data are collected, analyzed, and interpreted can affect what is discovered about human development across the lifespan.

Lifespan Development Copyright © 2020 by Julie Lazzara is licensed under a Creative Commons Attribution 4.0 International License , except where otherwise noted.

Share This Book

Development Research Methods

Note: This exam date is subjected to change based on seat availability. You can check final exam date on your hall ticket.

Page Visits

Course layout, books and references, instructor bio.

Prof. Rajshree Bedamatta

Course certificate.

DOWNLOAD APP

SWAYAM SUPPORT

Please choose the SWAYAM National Coordinator for support. * :

• Search Search Please fill out this field.
• What Is R&D?
• Understanding R&D
• Types of R&D
• Pros and Cons
• Considerations
• R&D vs. Applied Research
• Who Spends the Most?

The Bottom Line

• Business Essentials

Research and Development (R&D) Definition, Types, and Importance

Investopedia / Ellen Lindner

What Is Research and Development (R&D)?

The term research and development (R&D) is used to describe a series of activities that companies undertake to innovate and introduce new products and services. R&D is often the first stage in the development process. Companies require knowledge, talent, and investment in order to further their R&D needs and goals. The purpose of research and development is generally to take new products and services to market and add to the company's bottom line .

Key Takeaways

• Research and development represents the activities companies undertake to innovate and introduce new products and services or to improve their existing offerings.
• R&D allows a company to stay ahead of its competition by catering to new wants or needs in the market.
• Companies in different sectors and industries conduct R&D—pharmaceuticals, semiconductors, and technology companies generally spend the most.
• R&D is often a broad approach to exploratory advancement, while applied research is more geared towards researching a more narrow scope.
• The accounting for treatment for R&D costs can materially impact a company's income statement and balance sheet.

Understanding Research and Development (R&D)

The concept of research and development is widely linked to innovation both in the corporate and government sectors. R&D allows a company to stay ahead of its competition. Without an R&D program, a company may not survive on its own and may have to rely on other ways to innovate such as engaging in mergers and acquisitions (M&A) or partnerships. Through R&D, companies can design new products and improve their existing offerings.

R&D is distinct from most operational activities performed by a corporation. The research and/or development is typically not performed with the expectation of immediate profit. Instead, it is expected to contribute to the long-term profitability of a company. R&D may often allow companies to secure intellectual property, including patents , copyrights, and trademarks as discoveries are made and products created.

Companies that set up and employ departments dedicated entirely to R&D commit substantial capital to the effort. They must estimate the risk-adjusted return on their R&D expenditures, which inevitably involves risk of capital. That's because there is no immediate payoff, and the return on investment (ROI) is uncertain. As more money is invested in R&D, the level of capital risk increases. Other companies may choose to outsource their R&D for a variety of reasons including size and cost.

Companies across all sectors and industries undergo R&D activities. Corporations experience growth through these improvements and the development of new goods and services. Pharmaceuticals, semiconductors , and software/technology companies tend to spend the most on R&D. In Europe, R&D is known as research and technical or technological development.

Many small and mid-sized businesses may choose to outsource their R&D efforts because they don't have the right staff in-house to meet their needs.

Types of R&D

There are several different types of R&D that exist in the corporate world and within government. The type used depends entirely on the entity undertaking it and the results can differ.

Basic Research

There are business incubators and accelerators, where corporations invest in startups and provide funding assistance and guidance to entrepreneurs in the hope that innovations will result that they can use to their benefit.

M&As and partnerships are also forms of R&D as companies join forces to take advantage of other companies' institutional knowledge and talent.

Applied Research

One R&D model is a department staffed primarily by engineers who develop new products —a task that typically involves extensive research. There is no specific goal or application in mind with this model. Instead, the research is done for the sake of research.

Development Research

This model involves a department composed of industrial scientists or researchers, all of who are tasked with applied research in technical, scientific, or industrial fields. This model facilitates the development of future products or the improvement of current products and/or operating procedures.

$42.7 billion of research and development costs later, Amazon was granted 2,244 new patents in 2020. Their patents included advancements in artificial intelligence, machine learning, and cloud computing.

Advantages and Disadvantages of R&D

There are several key benefits to research and development. It facilitates innovation, allowing companies to improve existing products and services or by letting them develop new ones to bring to the market.

Because R&D also is a key component of innovation, it requires a greater degree of skill from employees who take part. This allows companies to expand their talent pool, which often comes with special skill sets.

The advantages go beyond corporations. Consumers stand to benefit from R&D because it gives them better, high-quality products and services as well as a wider range of options. Corporations can, therefore, rely on consumers to remain loyal to their brands. It also helps drive productivity and economic growth.

Disadvantages

One of the major drawbacks to R&D is the cost. First, there is the financial expense as it requires a significant investment of cash upfront. This can include setting up a separate R&D department, hiring talent, and product and service testing, among others.

Innovation doesn't happen overnight so there is also a time factor to consider. This means that it takes a lot of time to bring products and services to market from conception to production to delivery.

Because it does take time to go from concept to product, companies stand the risk of being at the mercy of changing market trends . So what they thought may be a great seller at one time may reach the market too late and not fly off the shelves once it's ready.

Facilitates innovation

Improved or new products and services

Expands knowledge and talent pool

Increased consumer choice and brand loyalty

Economic driver

Financial investment

Shifting market trends

R&D Accounting

R&D may be beneficial to a company's bottom line, but it is considered an expense . After all, companies spend substantial amounts on research and trying to develop new products and services. As such, these expenses are often reported for accounting purposes on the income statement and do not carry long-term value.

There are certain situations where R&D costs are capitalized and reported on the balance sheet. Some examples include but are not limited to:

• Materials, fixed assets, or other assets have alternative future uses with an estimable value and useful life.
• Software that can be converted or applied elsewhere in the company to have a useful life beyond a specific single R&D project.
• Indirect costs or overhead expenses allocated between projects.
• R&D purchased from a third party that is accompanied by intangible value. That intangible asset may be recorded as a separate balance sheet asset.

R&D Considerations

Before taking on the task of research and development, it's important for companies and governments to consider some of the key factors associated with it. Some of the most notable considerations are:

• Objectives and Outcome: One of the most important factors to consider is the intended goals of the R&D project. Is it to innovate and fill a need for certain products that aren't being sold? Or is it to make improvements on existing ones? Whatever the reason, it's always important to note that there should be some flexibility as things can change over time.
• Timing: R&D requires a lot of time. This involves reviewing the market to see where there may be a lack of certain products and services or finding ways to improve on those that are already on the shelves.
• Cost: R&D costs a great deal of money, especially when it comes to the upfront costs. And there may be higher costs associated with the conception and production of new products rather than updating existing ones.
• Risks: As with any venture, R&D does come with risks. R&D doesn't come with any guarantees, no matter the time and money that goes into it. This means that companies and governments may sacrifice their ROI if the end product isn't successful.

Research and Development vs. Applied Research

Basic research is aimed at a fuller, more complete understanding of the fundamental aspects of a concept or phenomenon. This understanding is generally the first step in R&D. These activities provide a basis of information without directed applications toward products, policies, or operational processes .

Applied research entails the activities used to gain knowledge with a specific goal in mind. The activities may be to determine and develop new products, policies, or operational processes. While basic research is time-consuming, applied research is painstaking and more costly because of its detailed and complex nature.

Who Spends the Most on R&D?

Companies spend billions of dollars on R&D to produce the newest, most sought-after products. According to public company filings, these companies incurred the highest research and development spending in 2020:

• Amazon: $42.7 billion
• Alphabet.: $27.6 billion
• Huawei: $22.0 billion
• Microsoft: $19.3 billion
• Apple: $18.8 billion
• Samsung: $18.8 billion
• Facebook: $18.5 billion

What Types of Activities Can Be Found in Research and Development?

Research and development activities focus on the innovation of new products or services in a company. Among the primary purposes of R&D activities is for a company to remain competitive as it produces products that advance and elevate its current product line. Since R&D typically operates on a longer-term horizon, its activities are not anticipated to generate immediate returns. However, in time, R&D projects may lead to patents, trademarks, or breakthrough discoveries with lasting benefits to the company. 

What Is an Example of Research and Development?

Alphabet allocated over $16 billion annually to R&D in 2018. Under its R&D arm X, the moonshot factory, it has developed Waymo self-driving cars. Meanwhile, Amazon has spent even more on R&D projects, with key developments in cloud computing and its cashier-less store Amazon Go. At the same time, R&D can take the approach of a merger & acquisition, where a company will leverage the talent and intel of another company to create a competitive edge. The same can be said with company investment in accelerators and incubators, whose developments it could later leverage.

Why Is Research and Development Important?

Given the rapid rate of technological advancement, R&D is important for companies to stay competitive. Specifically, R&D allows companies to create products that are difficult for their competitors to replicate. Meanwhile, R&D efforts can lead to improved productivity that helps increase margins, further creating an edge in outpacing competitors. From a broader perspective, R&D can allow a company to stay ahead of the curve, anticipating customer demands or trends.

There are many things companies can do in order to advance in their industries and the overall market. Research and development is just one way they can set themselves apart from their competition. It opens up the potential for innovation and increasing sales. But it does come with some drawbacks—the most obvious being the financial cost and the time it takes to innovate.

NASDAQ. " Which Companies Spend the Most in Research and Development (R&D)? "

Strategy+Business. " WHAT THE TOP INNOVATORS GET RIGHT ."

• Terms of Service
• Editorial Policy
• Privacy Policy
• Your Privacy Choices

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Language: English | German

How to Construct a Mixed Methods Research Design

Wie man ein mixed methods-forschungs-design konstruiert, judith schoonenboom.

1 Institut für Bildungswissenschaft, Universität Wien, Sensengasse 3a, 1090 Wien, Austria

R. Burke Johnson

2 Department of Professional Studies, University of South Alabama, UCOM 3700, 36688-0002 Mobile, AL USA

This article provides researchers with knowledge of how to design a high quality mixed methods research study. To design a mixed study, researchers must understand and carefully consider each of the dimensions of mixed methods design, and always keep an eye on the issue of validity. We explain the seven major design dimensions: purpose, theoretical drive, timing (simultaneity and dependency), point of integration, typological versus interactive design approaches, planned versus emergent design, and design complexity. There also are multiple secondary dimensions that need to be considered during the design process. We explain ten secondary dimensions of design to be considered for each research study. We also provide two case studies showing how the mixed designs were constructed.

Zusammenfassung

Der Beitrag gibt einen Überblick darüber, wie das Forschungsdesign bei Mixed Methods-Studien angelegt sein sollte. Um ein Mixed Methods-Forschungsdesign aufzustellen, müssen Forschende sorgfältig alle Dimensionen von Methodenkombinationen abwägen und von Anfang an auf die Güte und damit verbundene etwaige Probleme achten. Wir erklären und diskutieren die für Forschungsdesigns relevanten sieben Dimensionen von Methodenkombinationen: Untersuchungsziel, Rolle von Theorie im Forschungsprozess, Timing (Simultanität und Abhängigkeit), Schnittstellen, an denen Integration stattfindet, systematische vs. interaktive Design-Ansätze, geplante vs. emergente Designs und Komplexität des Designs. Es gibt außerdem zahlreiche sekundäre Dimensionen, die bei der Aufstellung des Forschungsdesigns berücksichtigt werden müssen, von denen wir zehn erklären. Der Beitrag schließt mit zwei Fallbeispielen ab, anhand derer konkret gezeigt wird, wie Mixed Methods-Forschungsdesigns aufgestellt werden können.

What is a mixed methods design?

This article addresses the process of selecting and constructing mixed methods research (MMR) designs. The word “design” has at least two distinct meanings in mixed methods research (Maxwell 2013 ). One meaning focuses on the process of design; in this meaning, design is often used as a verb. Someone can be engaged in designing a study (in German: “eine Studie konzipieren” or “eine Studie designen”). Another meaning is that of a product, namely the result of designing. The result of designing as a verb is a mixed methods design as a noun (in German: “das Forschungsdesign” or “Design”), as it has, for example, been described in a journal article. In mixed methods design, both meanings are relevant. To obtain a strong design as a product, one needs to carefully consider a number of rules for designing as an activity. Obeying these rules is not a guarantee of a strong design, but it does contribute to it. A mixed methods design is characterized by the combination of at least one qualitative and one quantitative research component. For the purpose of this article, we use the following definition of mixed methods research (Johnson et al. 2007 , p. 123):

Mixed methods research is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e. g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration.

Mixed methods research (“Mixed Methods” or “MM”) is the sibling of multimethod research (“Methodenkombination”) in which either solely multiple qualitative approaches or solely multiple quantitative approaches are combined.

In a commonly used mixed methods notation system (Morse 1991 ), the components are indicated as qual and quan (or QUAL and QUAN to emphasize primacy), respectively, for qualitative and quantitative research. As discussed below, plus (+) signs refer to concurrent implementation of components (“gleichzeitige Durchführung der Teilstudien” or “paralleles Mixed Methods-Design”) and arrows (→) refer to sequential implementation (“Sequenzielle Durchführung der Teilstudien” or “sequenzielles Mixed Methods-Design”) of components. Note that each research tradition receives an equal number of letters (four) in its abbreviation for equity. In this article, this notation system is used in some depth.

A mixed methods design as a product has several primary characteristics that should be considered during the design process. As shown in Table  1 , the following primary design “dimensions” are emphasized in this article: purpose of mixing, theoretical drive, timing, point of integration, typological use, and degree of complexity. These characteristics are discussed below. We also provide some secondary dimensions to consider when constructing a mixed methods design (Johnson and Christensen 2017 ).

List of Primary and Secondary Design Dimensions

On the basis of these dimensions, mixed methods designs can be classified into a mixed methods typology or taxonomy. In the mixed methods literature, various typologies of mixed methods designs have been proposed (for an overview see Creswell and Plano Clark 2011 , p. 69–72).

The overall goal of mixed methods research, of combining qualitative and quantitative research components, is to expand and strengthen a study’s conclusions and, therefore, contribute to the published literature. In all studies, the use of mixed methods should contribute to answering one’s research questions.

Ultimately, mixed methods research is about heightened knowledge and validity. The design as a product should be of sufficient quality to achieve multiple validities legitimation (Johnson and Christensen 2017 ; Onwuegbuzie and Johnson 2006 ), which refers to the mixed methods research study meeting the relevant combination or set of quantitative, qualitative, and mixed methods validities in each research study.

Given this goal of answering the research question(s) with validity, a researcher can nevertheless have various reasons or purposes for wanting to strengthen the research study and its conclusions. Following is the first design dimension for one to consider when designing a study: Given the research question(s), what is the purpose of the mixed methods study?

A popular classification of purposes of mixed methods research was first introduced in 1989 by Greene, Caracelli, and Graham, based on an analysis of published mixed methods studies. This classification is still in use (Greene 2007 ). Greene et al. ( 1989 , p. 259) distinguished the following five purposes for mixing in mixed methods research:

1.  Triangulation seeks convergence, corroboration, correspondence of results from different methods; 2.  Complementarity seeks elaboration, enhancement, illustration, clarification of the results from one method with the results from the other method; 3.  Development seeks to use the results from one method to help develop or inform the other method, where development is broadly construed to include sampling and implementation, as well as measurement decisions; 4.  Initiation seeks the discovery of paradox and contradiction, new perspectives of frameworks, the recasting of questions or results from one method with questions or results from the other method; 5.  Expansion seeks to extend the breadth and range of inquiry by using different methods for different inquiry components.

In the past 28 years, this classification has been supplemented by several others. On the basis of a review of the reasons for combining qualitative and quantitative research mentioned by the authors of mixed methods studies, Bryman ( 2006 ) formulated a list of more concrete rationales for performing mixed methods research (see Appendix). Bryman’s classification breaks down Greene et al.’s ( 1989 ) categories into several aspects, and he adds a number of additional aspects, such as the following:

(a)  Credibility – refers to suggestions that employing both approaches enhances the integrity of findings. (b)  Context – refers to cases in which the combination is justified in terms of qualitative research providing contextual understanding coupled with either generalizable, externally valid findings or broad relationships among variables uncovered through a survey. (c)  Illustration – refers to the use of qualitative data to illustrate quantitative findings, often referred to as putting “meat on the bones” of “dry” quantitative findings. (d)  Utility or improving the usefulness of findings – refers to a suggestion, which is more likely to be prominent among articles with an applied focus, that combining the two approaches will be more useful to practitioners and others. (e)  Confirm and discover – this entails using qualitative data to generate hypotheses and using quantitative research to test them within a single project. (f)  Diversity of views – this includes two slightly different rationales – namely, combining researchers’ and participants’ perspectives through quantitative and qualitative research respectively, and uncovering relationships between variables through quantitative research while also revealing meanings among research participants through qualitative research. (Bryman, p. 106)

Views can be diverse (f) in various ways. Some examples of mixed methods design that include a diversity of views are:

• Iteratively/sequentially connecting local/idiographic knowledge with national/general/nomothetic knowledge;
• Learning from different perspectives on teams and in the field and literature;
• Achieving multiple participation, social justice, and action;
• Determining what works for whom and the relevance/importance of context;
• Producing interdisciplinary substantive theory, including/comparing multiple perspectives and data regarding a phenomenon;
• Juxtaposition-dialogue/comparison-synthesis;
• Breaking down binaries/dualisms (some of both);
• Explaining interaction between/among natural and human systems;
• Explaining complexity.

The number of possible purposes for mixing is very large and is increasing; hence, it is not possible to provide an exhaustive list. Greene et al.’s ( 1989 ) purposes, Bryman’s ( 2006 ) rationales, and our examples of a diversity of views were formulated as classifications on the basis of examination of many existing research studies. They indicate how the qualitative and quantitative research components of a study relate to each other. These purposes can be used post hoc to classify research or a priori in the design of a new study. When designing a mixed methods study, it is sometimes helpful to list the purpose in the title of the study design.

The key point of this section is for the researcher to begin a study with at least one research question and then carefully consider what the purposes for mixing are. One can use mixed methods to examine different aspects of a single research question, or one can use separate but related qualitative and quantitative research questions. In all cases, the mixing of methods, methodologies, and/or paradigms will help answer the research questions and make improvements over a more basic study design. Fuller and richer information will be obtained in the mixed methods study.

Theoretical drive

In addition to a mixing purpose, a mixed methods research study might have an overall “theoretical drive” (Morse and Niehaus 2009 ). When designing a mixed methods study, it is occasionally helpful to list the theoretical drive in the title of the study design. An investigation, in Morse and Niehaus’s ( 2009 ) view, is focused primarily on either exploration-and-description or on testing-and-prediction. In the first case, the theoretical drive is called “inductive” or “qualitative”; in the second case, it is called “deductive” or “quantitative”. In the case of mixed methods, the component that corresponds to the theoretical drive is referred to as the “core” component (“Kernkomponente”), and the other component is called the “supplemental” component (“ergänzende Komponente”). In Morse’s notation system, the core component is written in capitals and the supplemental component is written in lowercase letters. For example, in a QUAL → quan design, more weight is attached to the data coming from the core qualitative component. Due to the decisive character of the core component, the core component must be able to stand on its own, and should be implemented rigorously. The supplemental component does not have to stand on its own.

Although this distinction is useful in some circumstances, we do not advise to apply it to every mixed methods design. First, Morse and Niehaus contend that the supplemental component can be done “less rigorously” but do not explain which aspects of rigor can be dropped. In addition, the idea of decreased rigor is in conflict with one key theme of the present article, namely that mixed methods designs should always meet the criterion of multiple validities legitimation (Onwuegbuzie and Johnson 2006 ).

The idea of theoretical drive as explicated by Morse and Niehaus has been criticized. For example, we view a theoretical drive as a feature not of a whole study, but of a research question, or, more precisely, of an interpretation of a research question. For example, if one study includes multiple research questions, it might include several theoretical drives (Schoonenboom 2016 ).

Another criticism of Morse and Niehaus’ conceptualization of theoretical drive is that it does not allow for equal-status mixed methods research (“Mixed Methods Forschung, bei der qualitative und quantitative Methoden die gleiche Bedeutung haben” or “gleichrangige Mixed Methods-Designs”), in which both the qualitative and quantitative component are of equal value and weight; this same criticism applies to Morgan’s ( 2014 ) set of designs. We agree with Greene ( 2015 ) that mixed methods research can be integrated at the levels of method, methodology, and paradigm. In this view, equal-status mixed methods research designs are possible, and they result when both the qualitative and the quantitative components, approaches, and thinking are of equal value, they take control over the research process in alternation, they are in constant interaction, and the outcomes they produce are integrated during and at the end of the research process. Therefore, equal-status mixed methods research (that we often advocate) is also called “interactive mixed methods research”.

Mixed methods research can have three different drives, as formulated by Johnson et al. ( 2007 , p. 123):

Qualitative dominant [or qualitatively driven] mixed methods research is the type of mixed research in which one relies on a qualitative, constructivist-poststructuralist-critical view of the research process, while concurrently recognizing that the addition of quantitative data and approaches are likely to benefit most research projects. Quantitative dominant [or quantitatively driven] mixed methods research is the type of mixed research in which one relies on a quantitative, postpositivist view of the research process, while concurrently recognizing that the addition of qualitative data and approaches are likely to benefit most research projects. (p. 124) The area around the center of the [qualitative-quantitative] continuum, equal status , is the home for the person that self-identifies as a mixed methods researcher. This researcher takes as his or her starting point the logic and philosophy of mixed methods research. These mixed methods researchers are likely to believe that qualitative and quantitative data and approaches will add insights as one considers most, if not all, research questions.

We leave it to the reader to decide if he or she desires to conduct a qualitatively driven study, a quantitatively driven study, or an equal-status/“interactive” study. According to the philosophies of pragmatism (Johnson and Onwuegbuzie 2004 ) and dialectical pluralism (Johnson 2017 ), interactive mixed methods research is very much a possibility. By successfully conducting an equal-status study, the pragmatist researcher shows that paradigms can be mixed or combined, and that the incompatibility thesis does not always apply to research practice. Equal status research is most easily conducted when a research team is composed of qualitative, quantitative, and mixed researchers, interacts continually, and conducts a study to address one superordinate goal.

Timing: simultaneity and dependence

Another important distinction when designing a mixed methods study relates to the timing of the two (or more) components. When designing a mixed methods study, it is usually helpful to include the word “concurrent” (“parallel”) or “sequential” (“sequenziell”) in the title of the study design; a complex design can be partially concurrent and partially sequential. Timing has two aspects: simultaneity and dependence (Guest 2013 ).

Simultaneity (“Simultanität”) forms the basis of the distinction between concurrent and sequential designs. In a  sequential design , the quantitative component precedes the qualitative component, or vice versa. In a  concurrent design , both components are executed (almost) simultaneously. In the notation of Morse ( 1991 ), concurrence is indicated by a “+” between components (e. g., QUAL + quan), while sequentiality is indicated with a “→” (QUAL → quan). Note that the use of capital letters for one component and lower case letters for another component in the same design suggest that one component is primary and the other is secondary or supplemental.

Some designs are sequential by nature. For example, in a  conversion design, qualitative categories and themes might be first obtained by collection and analysis of qualitative data, and then subsequently quantitized (Teddlie and Tashakkori 2009 ). Likewise, with Greene et al.’s ( 1989 ) initiation purpose, the initiation strand follows the unexpected results that it is supposed to explain. In other cases, the researcher has a choice. It is possible, e. g., to collect interview data and survey data of one inquiry simultaneously; in that case, the research activities would be concurrent. It is also possible to conduct the interviews after the survey data have been collected (or vice versa); in that case, research activities are performed sequentially. Similarly, a study with the purpose of expansion can be designed in which data on an effect and the intervention process are collected simultaneously, or they can be collected sequentially.

A second aspect of timing is dependence (“Abhängigkeit”) . We call two research components dependent if the implementation of the second component depends on the results of data analysis in the first component. Two research components are independent , if their implementation does not depend on the results of data analysis in the other component. Often, a researcher has a choice to perform data analysis independently or not. A researcher could analyze interview data and questionnaire data of one inquiry independently; in that case, the research activities would be independent. It is also possible to let the interview questions depend upon the outcomes of the analysis of the questionnaire data (or vice versa); in that case, research activities are performed dependently. Similarly, the empirical outcome/effect and process in a study with the purpose of expansion might be investigated independently, or the process study might take the effect/outcome as given (dependent).

In the mixed methods literature, the distinction between sequential and concurrent usually refers to the combination of concurrent/independent and sequential/dependent, and to the combination of data collection and data analysis. It is said that in a concurrent design, the data collection and data analysis of both components occurs (almost) simultaneously and independently, while in a sequential design, the data collection and data analysis of one component take place after the data collection and data analysis of the other component and depends on the outcomes of the other component.

In our opinion, simultaneity and dependence are two separate dimensions. Simultaneity indicates whether data collection is done concurrent or sequentially. Dependence indicates whether the implementation of one component depends upon the results of data analysis of the other component. As we will see in the example case studies, a concurrent design could include dependent data analysis, and a sequential design could include independent data analysis. It is conceivable that one simultaneously conducts interviews and collects questionnaire data (concurrent), while allowing the analysis focus of the interviews to depend on what emerges from the survey data (dependence).

Dependent research activities include a redirection of subsequent research inquiry. Using the outcomes of the first research component, the researcher decides what to do in the second component. Depending on the outcomes of the first research component, the researcher will do something else in the second component. If this is so, the research activities involved are said to be sequential-dependent, and any component preceded by another component should appropriately build on the previous component (see sequential validity legitimation ; Johnson and Christensen 2017 ; Onwuegbuzie and Johnson 2006 ).

It is under the purposive discretion of the researcher to determine whether a concurrent-dependent design, a concurrent-independent design, a sequential-dependent design, or a sequential-dependent design is needed to answer a particular research question or set of research questions in a given situation.

Point of integration

Each true mixed methods study has at least one “point of integration” – called the “point of interface” by Morse and Niehaus ( 2009 ) and Guest ( 2013 ) –, at which the qualitative and quantitative components are brought together. Having one or more points of integration is the distinguishing feature of a design based on multiple components. It is at this point that the components are “mixed”, hence the label “mixed methods designs”. The term “mixing”, however, is misleading, as the components are not simply mixed, but have to be integrated very carefully.

Determining where the point of integration will be, and how the results will be integrated, is an important, if not the most important, decision in the design of mixed methods research. Morse and Niehaus ( 2009 ) identify two possible points of integration: the results point of integration and the analytical point of integration.

Most commonly, integration takes place in the results point of integration . At some point in writing down the results of the first component, the results of the second component are added and integrated. A  joint display (listing the qualitative and quantitative findings and an integrative statement) might be used to facilitate this process.

In the case of an analytical point of integration , a first analytical stage of a qualitative component is followed by a second analytical stage, in which the topics identified in the first analytical stage are quantitized. The results of the qualitative component ultimately, and before writing down the results of the analytical phase as a whole, become quantitative; qualitizing also is a possible strategy, which would be the converse of this.

Other authors assume more than two possible points of integration. Teddlie and Tashakkori ( 2009 ) distinguish four different stages of an investigation: the conceptualization stage, the methodological experimental stage (data collection), the analytical experimental stage (data analysis), and the inferential stage. According to these authors, in all four stages, mixing is possible, and thus all four stages are potential points or integration.

However, the four possible points of integration used by Teddlie and Tashakkori ( 2009 ) are still too coarse to distinguish some types of mixing. Mixing in the experiential stage can take many different forms, for example the use of cognitive interviews to improve a questionnaire (tool development), or selecting people for an interview on the basis of the results of a questionnaire (sampling). Extending the definition by Guest ( 2013 ), we define the point of integration as “any point in a study where two or more research components are mixed or connected in some way”. Then, the point of integration in the two examples of this paragraph can be defined more accurately as “instrument development”, and “development of the sample”.

It is at the point of integration that qualitative and quantitative components are integrated. Some primary ways that the components can be connected to each other are as follows:

(1) merging the two data sets, (2) connecting from the analysis of one set of data to the collection of a second set of data, (3) embedding of one form of data within a larger design or procedure, and (4) using a framework (theoretical or program) to bind together the data sets (Creswell and Plano Clark 2011 , p. 76).

More generally, one can consider mixing at any or all of the following research components: purposes, research questions, theoretical drive, methods, methodology, paradigm, data, analysis, and results. One can also include mixing views of different researchers, participants, or stakeholders. The creativity of the mixed methods researcher designing a study is extensive.

Substantively, it can be useful to think of integration or mixing as comparing and bringing together two (or more) components on the basis of one or more of the purposes set out in the first section of this article. For example, it is possible to use qualitative data to illustrate a quantitative effect, or to determine whether the qualitative and the quantitative component yield convergent results ( triangulation ). An integrated result could also consist of a combination of a quantitatively established effect and a qualitative description of the underlying process . In the case of development, integration consists of an adjustment of an, often quantitative, for example, instrument or model or interpretation, based on qualitative assessments by members of the target group.

A special case is the integration of divergent results. The power of mixed methods research is its ability to deal with diversity and divergence. In the literature, we find two kinds of strategies for dealing with divergent results. A first set of strategies takes the detected divergence as the starting point for further analysis, with the aim to resolve the divergence. One possibility is to carry out further research (Cook 1985 ; Greene and Hall 2010 ). Further research is not always necessary. One can also look for a more comprehensive theory, which is able to account for both the results of the first component and the deviating results of the second component. This is a form of abduction (Erzberger and Prein 1997 ).

A fruitful starting point in trying to resolve divergence through abduction is to determine which component has resulted in a finding that is somehow expected, logical, and/or in line with existing research. The results of this research component, called the “sense” (“Lesart”), are subsequently compared to the results of the other component, called the “anti-sense” (“alternative Lesart”), which are considered dissonant, unexpected, and/or contrary to what had been found in the literature. The aim is to develop an overall explanation that fits both the sense and the anti-sense (Bazeley and Kemp 2012 ; Mendlinger and Cwikel 2008 ). Finally, a reanalysis of the data can sometimes lead to resolving divergence (Creswell and Plano Clark 2011 ).

Alternatively, one can question the existence of the encountered divergence. In this regard, Mathison ( 1988 ) recommends determining whether deviating results shown by the data can be explained by knowledge about the research and/or knowledge of the social world. Differences between results from different data sources could also be the result of properties of the methods involved, rather than reflect differences in reality (Yanchar and Williams 2006 ). In general, the conclusions of the individual components can be subjected to an inference quality audit (Teddlie and Tashakkori 2009 ), in which the researcher investigates the strength of each of the divergent conclusions. We recommend that researchers first determine whether there is “real” divergence, according to the strategies mentioned in the last paragraph. Next, an attempt can be made to resolve cases of “true” divergence, using one or more of the methods mentioned in this paragraph.

Design typology utilization

As already mentioned in Sect. 1, mixed methods designs can be classified into a mixed methods typology or taxonomy. A typology serves several purposes, including the following: guiding practice, legitimizing the field, generating new possibilities, and serving as a useful pedagogical tool (Teddlie and Tashakkori 2009 ). Note, however, that not all types of typologies are equally suitable for all purposes. For generating new possibilities, one will need a more exhaustive typology, while a useful pedagogical tool might be better served by a non-exhaustive overview of the most common mixed methods designs. Although some of the current MM design typologies include more designs than others, none of the current typologies is fully exhaustive. When designing a mixed methods study, it is often useful to borrow its name from an existing typology, or to construct a superior and nuanced clear name when your design is based on a modification of one or more of the designs.

Various typologies of mixed methods designs have been proposed. Creswell and Plano Clark’s ( 2011 ) typology of some “commonly used designs” includes six “major mixed methods designs”. Our summary of these designs runs as follows:

• Convergent parallel design (“paralleles Design”) (the quantitative and qualitative strands of the research are performed independently, and their results are brought together in the overall interpretation),
• Explanatory sequential design (“explanatives Design”) (a first phase of quantitative data collection and analysis is followed by the collection of qualitative data, which are used to explain the initial quantitative results),
• Exploratory sequential design (“exploratives Design”) (a first phase of qualitative data collection and analysis is followed by the collection of quantitative data to test or generalize the initial qualitative results),
• Embedded design (“Einbettungs-Design”) (in a traditional qualitative or quantitative design, a strand of the other type is added to enhance the overall design),
• Transformative design (“politisch-transformatives Design”) (a transformative theoretical framework, e. g. feminism or critical race theory, shapes the interaction, priority, timing and mixing of the qualitative and quantitative strand),
• Multiphase design (“Mehrphasen-Design”) (more than two phases or both sequential and concurrent strands are combined over a period of time within a program of study addressing an overall program objective).

Most of their designs presuppose a specific juxtaposition of the qualitative and quantitative component. Note that the last design is a complex type that is required in many mixed methods studies.

The following are our adapted definitions of Teddlie and Tashakkori’s ( 2009 ) five sets of mixed methods research designs (adapted from Teddlie and Tashakkori 2009 , p. 151):

• Parallel mixed designs (“paralleles Mixed-Methods-Design”) – In these designs, one has two or more parallel quantitative and qualitative strands, either with some minimal time lapse or simultaneously; the strand results are integrated into meta-inferences after separate analysis are conducted; related QUAN and QUAL research questions are answered or aspects of the same mixed research question is addressed.
• Sequential mixed designs (“sequenzielles Mixed-Methods-Design”) – In these designs, QUAL and QUAN strands occur across chronological phases, and the procedures/questions from the later strand emerge/depend/build on on the previous strand; the research questions are interrelated and sometimes evolve during the study.
• Conversion mixed designs (“Transfer-Design” or “Konversionsdesign”) – In these parallel designs, mixing occurs when one type of data is transformed to the other type and then analyzed, and the additional findings are added to the results; this design answers related aspects of the same research question,
• Multilevel mixed designs (“Mehrebenen-Mixed-Methods-Design”) – In these parallel or sequential designs, mixing occurs across multiple levels of analysis, as QUAN and QUAL data are analyzed and integrated to answer related aspects of the same research question or related questions.
• Fully integrated mixed designs (“voll integriertes Mixed-Methods-Design”) – In these designs, mixing occurs in an interactive manner at all stages of the study. At each stage, one approach affects the formulation of the other, and multiple types of implementation processes can occur. For example, rather than including integration only at the findings/results stage, or only across phases in a sequential design, mixing might occur at the conceptualization stage, the methodological stage, the analysis stage, and the inferential stage.

We recommend adding to Teddlie and Tashakkori’s typology a sixth design type, specifically, a  “hybrid” design type to include complex combinations of two or more of the other design types. We expect that many published MM designs will fall into the hybrid design type.

Morse and Niehaus ( 2009 ) listed eight mixed methods designs in their book (and suggested that authors create more complex combinations when needed). Our shorthand labels and descriptions (adapted from Morse and Niehaus 2009 , p. 25) run as follows:

• QUAL + quan (inductive-simultaneous design where, the core component is qualitative and the supplemental component is quantitative)
• QUAL → quan (inductive-sequential design, where the core component is qualitative and the supplemental component is quantitative)
• QUAN + qual (deductive-simultaneous design where, the core component is quantitative and the supplemental component is qualitative)
• QUAN → qual (deductive-sequential design, where the core component is quantitative and the supplemental component is qualitative)
• QUAL + qual (inductive-simultaneous design, where both components are qualitative; this is a multimethod design rather than a mixed methods design)
• QUAL → qual (inductive-sequential design, where both components are qualitative; this is a multimethod design rather than a mixed methods design)
• QUAN + quan (deductive-simultaneous design, where both components are quantitative; this is a multimethod design rather than a mixed methods design)
• QUAN → quan (deductive-sequential design, where both components are quantitative; this is a multimethod design rather than a mixed methods design).

Notice that Morse and Niehaus ( 2009 ) included four mixed methods designs (the first four designs shown above) and four multimethod designs (the second set of four designs shown above) in their typology. The reader can, therefore, see that the design notation also works quite well for multimethod research designs. Notably absent from Morse and Niehaus’s book are equal-status or interactive designs. In addition, they assume that the core component should always be performed either concurrent with or before the supplemental component.

Johnson, Christensen, and Onwuegbuzie constructed a set of mixed methods designs without these limitations. The resulting mixed methods design matrix (see Johnson and Christensen 2017 , p. 478) contains nine designs, which we can label as follows (adapted from Johnson and Christensen 2017 , p. 478):

• QUAL + QUAN (equal-status concurrent design),
• QUAL + quan (qualitatively driven concurrent design),
• QUAN + qual (quantitatively driven concurrent design),
• QUAL → QUAN (equal-status sequential design),
• QUAN → QUAL (equal-status sequential design),
• QUAL → quan (qualitatively driven sequential design),
• qual → QUAN (quantitatively driven sequential design),
• QUAN → qual (quantitatively driven sequential design), and
• quan → QUAL (qualitatively driven sequential design).

The above set of nine designs assumed only one qualitative and one quantitative component. However, this simplistic assumption can be relaxed in practice, allowing the reader to construct more complex designs. The Morse notation system is very powerful. For example, here is a three-stage equal-status concurrent-sequential design:

The key point here is that the Morse notation provides researchers with a powerful language for depicting and communicating the design constructed for a specific research study.

When designing a mixed methods study, it is sometimes helpful to include the mixing purpose (or characteristic on one of the other dimensions shown in Table  1 ) in the title of the study design (e. g., an explanatory sequential MM design, an exploratory-confirmatory MM design, a developmental MM design). Much more important, however, than a design name is for the author to provide an accurate description of what was done in the research study, so the reader will know exactly how the study was conducted. A design classification label can never replace such a description.

The common complexity of mixed methods design poses a problem to the above typologies of mixed methods research. The typologies were designed to classify whole mixed methods studies, and they are basically based on a classification of simple designs. In practice, many/most designs are complex. Complex designs are sometimes labeled “complex design”, “multiphase design”, “fully integrated design”, “hybrid design” and the like. Because complex designs occur very often in practice, the above typologies are not able to classify a large part of existing mixed methods research any further than by labeling them “complex”, which in itself is not very informative about the particular design. This problem does not fully apply to Morse’s notation system, which can be used to symbolize some more complex designs.

Something similar applies to the classification of the purposes of mixed methods research. The classifications of purposes mentioned in the “Purpose”-section, again, are basically meant for the classification of whole mixed methods studies. In practice, however, one single study often serves more than one purpose (Schoonenboom et al. 2017 ). The more purposes that are included in one study, the more difficult it becomes to select a design on the basis of the purpose of the investigation, as advised by Greene ( 2007 ). Of all purposes involved, then, which one should be the primary basis for the design? Or should the design be based upon all purposes included? And if so, how? For more information on how to articulate design complexity based on multiple purposes of mixing, see Schoonenboom et al. ( 2017 ).

It should be clear to the reader that, although much progress has been made in the area of mixed methods design typologies, the problem remains in developing a single typology that is effective in comprehensively listing a set of designs for mixed methods research. This is why we emphasize in this article the importance of learning to build on simple designs and construct one’s own design for one’s research questions. This will often result in a combination or “hybrid” design that goes beyond basic designs found in typologies, and a methodology section that provides much more information than a design name.

Typological versus interactive approaches to design

In the introduction, we made a distinction between design as a product and design as a process. Related to this, two different approaches to design can be distinguished: typological/taxonomic approaches (“systematische Ansätze”), such as those in the previous section, and interactive approaches (“interaktive Ansätze”) (the latter were called “dynamic” approaches by Creswell and Plano Clark 2011 ). Whereas typological/taxonomic approaches view designs as a sort of mold, in which the inquiry can be fit, interactive approaches (Maxwell 2013 ) view design as a process, in which a certain design-as-a-product might be the outcome of the process, but not its input.

The most frequently mentioned interactive approach to mixed methods research is the approach by Maxwell and Loomis ( 2003 ). Maxwell and Loomis distinguish the following components of a design: goals, conceptual framework, research question, methods, and validity. They argue convincingly that the most important task of the researcher is to deliver as the end product of the design process a design in which these five components fit together properly. During the design process, the researcher works alternately on the individual components, and as a result, their initial fit, if it existed, tends to get lost. The researcher should therefore regularly check during the research and continuing design process whether the components still fit together, and, if not, should adapt one or the other component to restore the fit between them. In an interactive approach, unlike the typological approach, design is viewed as an interactive process in which the components are continually compared during the research study to each other and adapted to each other.

Typological and interactive approaches to mixed methods research have been presented as mutually exclusive alternatives. In our view, however, they are not mutually exclusive. The interactive approach of Maxwell is a very powerful tool for conducting research, yet this approach is not specific to mixed methods research. Maxwell’s interactive approach emphasizes that the researcher should keep and monitor a close fit between the five components of research design. However, it does not indicate how one should combine qualitative and quantitative subcomponents within one of Maxwell’s five components (e. g., how one should combine a qualitative and a quantitative method, or a qualitative and a quantitative research question). Essential elements of the design process, such as timing and the point of integration are not covered by Maxwell’s approach. This is not a shortcoming of Maxwell’s approach, but it indicates that to support the design of mixed methods research, more is needed than Maxwell’s model currently has to offer.

Some authors state that design typologies are particularly useful for beginning researchers and interactive approaches are suited for experienced researchers (Creswell and Plano Clark 2011 ). However, like an experienced researcher, a research novice needs to align the components of his or her design properly with each other, and, like a beginning researcher, an advanced researcher should indicate how qualitative and quantitative components are combined with each other. This makes an interactive approach desirable, also for beginning researchers.

We see two merits of the typological/taxonomic approach . We agree with Greene ( 2007 ), who states that the value of the typological approach mainly lies in the different dimensions of mixed methods that result from its classifications. In this article, the primary dimensions include purpose, theoretical drive, timing, point of integration, typological vs. interactive approaches, planned vs. emergent designs, and complexity (also see secondary dimensions in Table  1 ). Unfortunately, all of these dimensions are not reflected in any single design typology reviewed here. A second merit of the typological approach is the provision of common mixed methods research designs, of common ways in which qualitative and quantitative research can be combined, as is done for example in the major designs of Creswell and Plano Clark ( 2011 ). Contrary to other authors, however, we do not consider these designs as a feature of a whole study, but rather, in line with Guest ( 2013 ), as a feature of one part of a design in which one qualitative and one quantitative component are combined. Although one study could have only one purpose, one point of integration, et cetera, we believe that combining “designs” is the rule and not the exception. Therefore, complex designs need to be constructed and modified as needed, and during the writing phase the design should be described in detail and perhaps given a creative and descriptive name.

Planned versus emergent designs

A mixed methods design can be thought out in advance, but can also arise during the course of the conduct of the study; the latter is called an “emergent” design (Creswell and Plano Clark 2011 ). Emergent designs arise, for example, when the researcher discovers during the study that one of the components is inadequate (Morse and Niehaus 2009 ). Addition of a component of the other type can sometimes remedy such an inadequacy. Some designs contain an emergent component by their nature. Initiation, for example, is the further exploration of unexpected outcomes. Unexpected outcomes are by definition not foreseen, and therefore cannot be included in the design in advance.

The question arises whether researchers should plan all these decisions beforehand, or whether they can make them during, and depending on the course of, the research process. The answer to this question is twofold. On the one hand, a researcher should decide beforehand which research components to include in the design, such that the conclusion that will be drawn will be robust. On the other hand, developments during research execution will sometimes prompt the researcher to decide to add additional components. In general, the advice is to be prepared for the unexpected. When one is able to plan for emergence, one should not refrain from doing so.

Dimension of complexity

Next, mixed methods designs are characterized by their complexity. In the literature, simple and complex designs are distinguished in various ways. A common distinction is between simple investigations with a single point of integration versus complex investigations with multiple points of integration (Guest 2013 ). When designing a mixed methods study, it can be useful to mention in the title whether the design of the study is simple or complex. The primary message of this section is as follows: It is the responsibility of the researcher to create more complex designs when needed to answer his or her research question(s) .

Teddlie and Tashakkori’s ( 2009 ) multilevel mixed designs and fully integrated mixed designs are both complex designs, but for different reasons. A multilevel mixed design is more complex ontologically, because it involves multiple levels of reality. For example, data might be collected both at the levels of schools and students, neighborhood and households, companies and employees, communities and inhabitants, or medical practices and patients (Yin 2013 ). Integration of these data does not only involve the integration of qualitative and quantitative data, but also the integration of data originating from different sources and existing at different levels. Little if any published research has discussed the possible ways of integrating data obtained in a multilevel mixed design (see Schoonenboom 2016 ). This is an area in need of additional research.

The fully-integrated mixed design is more complex because it contains multiple points of integration. As formulated by Teddlie and Tashakkori ( 2009 , p. 151):

In these designs, mixing occurs in an interactive manner at all stages of the study. At each stage, one approach affects the formulation of the other, and multiple types of implementation processes can occur.

Complexity, then, not only depends on the number of components, but also on the extent to which they depend on each other (e. g., “one approach affects the formulation of the other”).

Many of our design dimensions ultimately refer to different ways in which the qualitative and quantitative research components are interdependent. Different purposes of mixing ultimately differ in the way one component relates to, and depends upon, the other component. For example, these purposes include dependencies, such as “x illustrates y” and “x explains y”. Dependencies in the implementation of x and y occur to the extent that the design of y depends on the results of x (sequentiality). The theoretical drive creates dependencies, because the supplemental component y is performed and interpreted within the context and the theoretical drive of core component x. As a general rule in designing mixed methods research, one should examine and plan carefully the ways in which and the extent to which the various components depend on each other.

The dependence among components, which may or may not be present, has been summarized by Greene ( 2007 ). It is seen in the distinction between component designs (“Komponenten-Designs”), in which the components are independent of each other, and integrated designs (“integrierte Designs”), in which the components are interdependent. Of these two design categories, integrated designs are the more complex designs.

Secondary design considerations

The primary design dimensions explained above have been the focus of this article. There are a number of secondary considerations for researchers to also think about when they design their studies (Johnson and Christensen 2017 ). Now we list some secondary design issues and questions that should be thoughtfully considered during the construction of a strong mixed methods research design.

• Phenomenon: Will the study be addressing (a) the same part or different parts of one phenomenon? (b) different phenomena?, or (c) the phenomenon/phenomena from different perspectives? Is the phenomenon (a) expected to be unique (e. g., historical event, particular group)?, (b) something expected to be part of a more regular and predictable phenomenon, or (c) a complex mixture of these?
• Social scientific theory: Will the study generate a new substantive theory, test an already constructed theory, or achieve both in a sequential arrangement? Or is the researcher not interested in substantive theory based on empirical data?
• Ideological drive: Will the study have an explicitly articulated ideological drive (e. g., feminism, critical race paradigm, transformative paradigm)?
• Combination of sampling methods: What specific quantitative sampling method(s) will be used? What specific qualitative sampling methods(s) will be used? How will these be combined or related?
• Degree to which the research participants will be similar or different: For example, participants or stakeholders with known differences of perspective would provide participants that are quite different.
• Degree to which the researchers on the research team will be similar or different: For example, an experiment conducted by one researcher would be high on similarity, but the use of a heterogeneous and participatory research team would include many differences.
• Implementation setting: Will the phenomenon be studied naturalistically, experimentally, or through a combination of these?
• Degree to which the methods similar or different: For example, a structured interview and questionnaire are fairly similar but administration of a standardized test and participant observation in the field are quite different.
• Validity criteria and strategies: What validity criteria and strategies will be used to address the defensibility of the study and the conclusions that will be drawn from it (see Chapter 11 in Johnson and Christensen 2017 )?
• Full study: Will there be essentially one research study or more than one? How will the research report be structured?

Two case studies

The above design dimensions are now illustrated by examples. A nice collection of examples of mixed methods studies can be found in Hesse-Biber ( 2010 ), from which the following examples are taken. The description of the first case example is shown in Box 1.

Box 1

Summary of Roth ( 2006 ), research regarding the gender-wage gap within Wall Street securities firms. Adapted from Hesse-Biber ( 2010 , pp. 457–458)

Louise Marie Roth’s research, Selling Women Short: Gender and Money on Wall Street ( 2006 ), tackles gender inequality in the workplace. She was interested in understanding the gender-wage gap among highly performing Wall Street MBAs, who on the surface appeared to have the same “human capital” qualifications and were placed in high-ranking Wall Street securities firms as their first jobs. In addition, Roth wanted to understand the “structural factors” within the workplace setting that may contribute to the gender-wage gap and its persistence over time. […] Roth conducted semistructured interviews, nesting quantitative closed-ended questions into primarily qualitative in-depth interviews […] In analyzing the quantitative data from her sample, she statistically considered all those factors that might legitimately account for gendered differences such as number of hours worked, any human capital differences, and so on. Her analysis of the quantitative data revealed the presence of a significant gender gap in wages that remained unexplained after controlling for any legitimate factors that might otherwise make a difference. […] Quantitative findings showed the extent of the wage gap while providing numerical understanding of the disparity but did not provide her with an understanding of the specific processes within the workplace that might have contributed to the gender gap in wages. […] Her respondents’ lived experiences over time revealed the hidden inner structures of the workplace that consist of discriminatory organizational practices with regard to decision making in performance evaluations that are tightly tied to wage increases and promotion.

This example nicely illustrates the distinction we made between simultaneity and dependency. On the two aspects of the timing dimension, this study was a concurrent-dependent design answering a set of related research questions. The data collection in this example was conducted simultaneously, and was thus concurrent – the quantitative closed-ended questions were embedded into the qualitative in-depth interviews. In contrast, the analysis was dependent, as explained in the next paragraph.

One of the purposes of this study was explanation: The qualitative data were used to understand the processes underlying the quantitative outcomes. It is therefore an explanatory design, and might be labelled an “explanatory concurrent design”. Conceptually, explanatory designs are often dependent: The qualitative component is used to explain and clarify the outcomes of the quantitative component. In that sense, the qualitative analysis in the case study took the outcomes of the quantitative component (“the existence of the gender-wage gap” and “numerical understanding of the disparity”), and aimed at providing an explanation for that result of the quantitative data analysis , by relating it to the contextual circumstances in which the quantitative outcomes were produced. This purpose of mixing in the example corresponds to Bryman’s ( 2006 ) “contextual understanding”. On the other primary dimensions, (a) the design was ongoing over a three-year period but was not emergent, (b) the point of integration was results, and (c) the design was not complex with respect to the point of integration, as it had only one point of integration. Yet, it was complex in the sense of involving multiple levels; both the level of the individual and the organization were included. According to the approach of Johnson and Christensen ( 2017 ), this was a QUAL + quan design (that was qualitatively driven, explanatory, and concurrent). If we give this study design a name, perhaps it should focus on what was done in the study: “explaining an effect from the process by which it is produced”. Having said this, the name “explanatory concurrent design” could also be used.

The description of the second case example is shown in Box 2.

Box 2

Summary of McMahon’s ( 2007 ) explorative study of the meaning, role, and salience of rape myths within the subculture of college student athletes. Adapted from Hesse-Biber ( 2010 , pp. 461–462)

Sarah McMahon ( 2007 ) wanted to explore the subculture of college student athletes and specifically the meaning, role, and salience of rape myths within that culture. […] While she was looking for confirmation between the quantitative ([structured] survey) and qualitative (focus groups and individual interviews) findings, she entered this study skeptical of whether or not her quantitative and qualitative findings would mesh with one another. McMahon […] first administered a survey [instrument] to 205 sophomore and junior student athletes at one Northeast public university. […] The quantitative data revealed a very low acceptance of rape myths among this student population but revealed a higher acceptance of violence among men and individuals who did not know a survivor of sexual assault. In the second qualitative (QUAL) phase, “focus groups were conducted as semi-structured interviews” and facilitated by someone of the same gender as the participants (p. 360). […] She followed this up with a third qualitative component (QUAL), individual interviews, which were conducted to elaborate on themes discovered in the focus groups and determine any differences in students’ responses between situations (i. e., group setting vs. individual). The interview guide was designed specifically to address focus group topics that needed “more in-depth exploration” or clarification (p. 361). The qualitative findings from the focus groups and individual qualitative interviews revealed “subtle yet pervasive rape myths” that fell into four major themes: “the misunderstanding of consent, the belief in ‘accidental’ and fabricated rape, the contention that some women provoke rape, and the invulnerability of female athletes” (p. 363). She found that the survey’s finding of a “low acceptance of rape myths … was contradicted by the findings of the focus groups and individual interviews, which indicated the presence of subtle rape myths” (p. 362).

On the timing dimension, this is an example of a sequential-independent design. It is sequential, because the qualitative focus groups were conducted after the survey was administered. The analysis of the quantitative and qualitative data was independent: Both were analyzed independently, to see whether they yielded the same results (which they did not). This purpose, therefore, was triangulation. On the other primary dimensions, (a) the design was planned, (b) the point of integration was results, and (c) the design was not complex as it had only one point of integration, and involved only the level of the individual. The author called this a “sequential explanatory” design. We doubt, however, whether this is the most appropriate label, because the qualitative component did not provide an explanation for quantitative results that were taken as given. On the contrary, the qualitative results contradicted the quantitative results. Thus, a “sequential-independent” design, or a “sequential-triangulation” design or a “sequential-comparative” design would probably be a better name.

Notice further that the second case study had the same point of integration as the first case study. The two components were brought together in the results. Thus, although the case studies are very dissimilar in many respects, this does not become visible in their point of integration. It can therefore be helpful to determine whether their point of extension is different. A  point of extension is the point in the research process at which the second (or later) component comes into play. In the first case study, two related, but different research questions were answered, namely the quantitative question “How large is the gender-wage gap among highly performing Wall Street MBAs after controlling for any legitimate factors that might otherwise make a difference?”, and the qualitative research question “How do structural factors within the workplace setting contribute to the gender-wage gap and its persistence over time?” This case study contains one qualitative research question and one quantitative research question. Therefore, the point of extension is the research question. In the second case study, both components answered the same research question. They differed in their data collection (and subsequently in their data analysis): qualitative focus groups and individual interviews versus a quantitative questionnaire. In this case study, the point of extension was data collection. Thus, the point of extension can be used to distinguish between the two case studies.

Summary and conclusions

The purpose of this article is to help researchers to understand how to design a mixed methods research study. Perhaps the simplest approach is to design is to look at a single book and select one from the few designs included in that book. We believe that is only useful as a starting point. Here we have shown that one often needs to construct a research design to fit one’s unique research situation and questions.

First, we showed that there are there are many purposes for which qualitative and quantitative methods, methodologies, and paradigms can be mixed. This must be determined in interaction with the research questions. Inclusion of a purpose in the design name can sometimes provide readers with useful information about the study design, as in, e. g., an “explanatory sequential design” or an “exploratory-confirmatory design”.

The second dimension is theoretical drive in the sense that Morse and Niehaus ( 2009 ) use this term. That is, will the study have an inductive or a deductive drive, or, we added, a combination of these. Related to this idea is whether one will conduct a qualitatively driven, a quantitatively driven, or an equal-status mixed methods study. This language is sometimes included in the design name to communicate this characteristic of the study design (e. g., a “quantitatively driven sequential mixed methods design”).

The third dimension is timing , which has two aspects: simultaneity and dependence. Simultaneity refers to whether the components are to be implemented concurrently, sequentially, or a combination of these in a multiphase design. Simultaneity is commonly used in the naming of a mixed methods design because it communicates key information. The second aspect of timing, dependence , refers to whether a later component depends on the results of an earlier component, e. g., Did phase two specifically build on phase one in the research study? The fourth design dimension is the point of integration, which is where the qualitative and quantitative components are brought together and integrated. This is an essential dimension, but it usually does not need to be incorporated into the design name.

The fifth design dimension is that of typological vs. interactive design approaches . That is, will one select a design from a typology or use a more interactive approach to construct one’s own design? There are many typologies of designs currently in the literature. Our recommendation is that readers examine multiple design typologies to better understand the design process in mixed methods research and to understand what designs have been identified as popular in the field. However, when a design that would follow from one’s research questions is not available, the researcher can and should (a) combine designs into new designs or (b) simply construct a new and unique design. One can go a long way in depicting a complex design with Morse’s ( 1991 ) notation when used to its full potential. We also recommend that researchers understand the process approach to design from Maxwell and Loomis ( 2003 ), and realize that research design is a process and it needs, oftentimes, to be flexible and interactive.

The sixth design dimension or consideration is whether a design will be fully specified during the planning of the research study or if the design (or part of the design) will be allowed to emerge during the research process, or a combination of these. The seventh design dimension is called complexity . One sort of complexity mentioned was multilevel designs, but there are many complexities that can enter designs. The key point is that good research often requires the use of complex designs to answer one’s research questions. This is not something to avoid. It is the responsibility of the researcher to learn how to construct and describe and name mixed methods research designs. Always remember that designs should follow from one’s research questions and purposes, rather than questions and purposes following from a few currently named designs.

In addition to the six primary design dimensions or considerations, we provided a set of additional or secondary dimensions/considerations or questions to ask when constructing a mixed methods study design. Our purpose throughout this article has been to show what factors must be considered to design a high quality mixed methods research study. The more one knows and thinks about the primary and secondary dimensions of mixed methods design the better equipped one will be to pursue mixed methods research.

Acknowledgments

Open access funding provided by University of Vienna.

Biographies

1965, Dr., Professor of Empirical Pedagogy at University of Vienna, Austria. Research Areas: Mixed Methods Design, Philosophy of Mixed Methods Research, Innovation in Higher Education, Design and Evaluation of Intervention Studies, Educational Technology. Publications: Mixed methods in early childhood education. In: M. Fleer & B. v. Oers (Eds.), International handbook on early childhood education (Vol. 1). Dordrecht, The Netherlands: Springer 2017; The multilevel mixed intact group analysis: A mixed method to seek, detect, describe and explain differences between intact groups. Journal of Mixed Methods Research 10, 2016; The realist survey: How respondents’ voices can be used to test and revise correlational models. Journal of Mixed Methods Research 2015. Advance online publication.

1957, PhD, Professor of Professional Studies at University of South Alabama, Mobile, Alabama USA. Research Areas: Methods of Social Research, Program Evaluation, Quantitative, Qualitative and Mixed Methods, Philosophy of Social Science. Publications: Research methods, design and analysis. Boston, MA 2014 (with L. Christensen and L. Turner); Educational research: Quantitative, qualitative and mixed approaches. Los Angeles, CA 2017 (with L. Christensen); The Oxford handbook of multimethod and mixed methods research inquiry. New York, NY 2015 (with S. Hesse-Biber).

Bryman’s ( 2006 ) scheme of rationales for combining quantitative and qualitative research 1

• Triangulation or greater validity – refers to the traditional view that quantitative and qualitative research might be combined to triangulate findings in order that they may be mutually corroborated. If the term was used as a synonym for integrating quantitative and qualitative research, it was not coded as triangulation.
• Offset – refers to the suggestion that the research methods associated with both quantitative and qualitative research have their own strengths and weaknesses so that combining them allows the researcher to offset their weaknesses to draw on the strengths of both.
• Completeness – refers to the notion that the researcher can bring together a more comprehensive account of the area of enquiry in which he or she is interested if both quantitative and qualitative research are employed.
• Process – quantitative research provides an account of structures in social life but qualitative research provides sense of process.
• Different research questions – this is the argument that quantitative and qualitative research can each answer different research questions but this item was coded only if authors explicitly stated that they were doing this.
• Explanation – one is used to help explain findings generated by the other.
• Unexpected results – refers to the suggestion that quantitative and qualitative research can be fruitfully combined when one generates surprising results that can be understood by employing the other.
• Instrument development – refers to contexts in which qualitative research is employed to develop questionnaire and scale items – for example, so that better wording or more comprehensive closed answers can be generated.
• Sampling – refers to situations in which one approach is used to facilitate the sampling of respondents or cases.
• Credibility – refer s to suggestions that employing both approaches enhances the integrity of findings.
• Context – refers to cases in which the combination is rationalized in terms of qualitative research providing contextual understanding coupled with either generalizable, externally valid findings or broad relationships among variables uncovered through a survey.
• Illustration – refers to the use of qualitative data to illustrate quantitative findings, often referred to as putting “meat on the bones” of “dry” quantitative findings.
• Utility or improving the usefulness of findings – refers to a suggestion, which is more likely to be prominent among articles with an applied focus, that combining the two approaches will be more useful to practitioners and others.
• Confirm and discover – this entails using qualitative data to generate hypotheses and using quantitative research to test them within a single project.
• Diversity of views – this includes two slightly different rationales – namely, combining researchers’ and participants’ perspectives through quantitative and qualitative research respectively, and uncovering relationships between variables through quantitative research while also revealing meanings among research participants through qualitative research.
• Enhancement or building upon quantitative/qualitative findings – this entails a reference to making more of or augmenting either quantitative or qualitative findings by gathering data using a qualitative or quantitative research approach.
• Other/unclear.
• Not stated.

1 Reprinted with permission from “Integrating quantitative and qualitative research: How is it done?” by Alan Bryman ( 2006 ), Qualitative Research, 6, pp. 105–107.

Contributor Information

Judith Schoonenboom, Email: [email protected] .

R. Burke Johnson, Email: ude.amabalahtuos@nosnhojb .

• Bazeley, Pat, Lynn Kemp Mosaics, triangles, and DNA: Metaphors for integrated analysis in mixed methods research. Journal of Mixed Methods Research. 2012; 6 :55–72. doi: 10.1177/1558689811419514. [ CrossRef ] [ Google Scholar ]
• Bryman A. Integrating quantitative and qualitative research: how is it done? Qualitative Research. 2006; 6 :97–113. doi: 10.1177/1468794106058877. [ CrossRef ] [ Google Scholar ]
• Cook TD. Postpositivist critical multiplism. In: Shotland RL, Mark MM, editors. Social science and social policy. Beverly Hills: SAGE; 1985. pp. 21–62. [ Google Scholar ]
• Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 2. Los Angeles: SAGE; 2011. [ Google Scholar ]
• Erzberger C, Prein G. Triangulation: Validity and empirically-based hypothesis construction. Quality and Quantity. 1997; 31 :141–154. doi: 10.1023/A:1004249313062. [ CrossRef ] [ Google Scholar ]
• Greene JC. Mixed methods in social inquiry. San Francisco: Jossey-Bass; 2007. [ Google Scholar ]
• Greene JC. Preserving distinctions within the multimethod and mixed methods research merger. Sharlene Hesse-Biber and R. Burke Johnson. New York: Oxford University Press; 2015. [ Google Scholar ]
• Greene JC, Valerie J, Caracelli, Graham WF. Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis. 1989; 11 :255–274. doi: 10.3102/01623737011003255. [ CrossRef ] [ Google Scholar ]
• Greene JC, Hall JN. Dialectics and pragmatism. In: Tashakkori A, Teddlie C, editors. SAGE handbook of mixed methods in social & behavioral research. 2. Los Angeles: SAGE; 2010. pp. 119–167. [ Google Scholar ]
• Guest, Greg Describing mixed methods research: An alternative to typologies. Journal of Mixed Methods Research. 2013; 7 :141–151. doi: 10.1177/1558689812461179. [ CrossRef ] [ Google Scholar ]
• Hesse-Biber S. Qualitative approaches to mixed methods practice. Qualitative Inquiry. 2010; 16 :455–468. doi: 10.1177/1077800410364611. [ CrossRef ] [ Google Scholar ]
• Johnson BR. Dialectical pluralism: A metaparadigm whose time has come. Journal of Mixed Methods Research. 2017; 11 :156–173. doi: 10.1177/1558689815607692. [ CrossRef ] [ Google Scholar ]
• Johnson BR, Christensen LB. Educational research: Quantitative, qualitative, and mixed approaches. 6. Los Angeles: SAGE; 2017. [ Google Scholar ]
• Johnson BR, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educational Researcher. 2004; 33 (7):14–26. doi: 10.3102/0013189X033007014. [ CrossRef ] [ Google Scholar ]
• Johnson BR, Onwuegbuzie AJ, Turner LA. Toward a definition of mixed methods research. Journal of Mixed Methods Research. 2007; 1 :112–133. doi: 10.1177/1558689806298224. [ CrossRef ] [ Google Scholar ]
• Mathison S. Why triangulate? Educational Researcher. 1988; 17 :13–17. doi: 10.3102/0013189X017002013. [ CrossRef ] [ Google Scholar ]
• Maxwell JA. Qualitative research design: An interactive approach. 3. Los Angeles: SAGE; 2013. [ Google Scholar ]
• Maxwell, Joseph A., and Diane M. Loomis. 2003. Mixed methods design: An alternative approach. In Handbook of mixed methods in social & behavioral research , Eds. Abbas Tashakkori and Charles Teddlie, 241–271. Thousand Oaks: Sage.
• McMahon S. Understanding community-specific rape myths: Exploring student athlete culture. Affilia. 2007; 22 :357–370. doi: 10.1177/0886109907306331. [ CrossRef ] [ Google Scholar ]
• Mendlinger S, Cwikel J. Spiraling between qualitative and quantitative data on women’s health behaviors: A double helix model for mixed methods. Qualitative Health Research. 2008; 18 :280–293. doi: 10.1177/1049732307312392. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Morgan DL. Integrating qualitative and quantitative methods: a pragmatic approach. Los Angeles: Sage; 2014. [ Google Scholar ]
• Morse JM. Approaches to qualitative-quantitative methodological triangulation. Nursing Research. 1991; 40 :120–123. doi: 10.1097/00006199-199103000-00014. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Morse JM, Niehaus L. Mixed method design: Principles and procedures. Walnut Creek: Left Coast Press; 2009. [ Google Scholar ]
• Onwuegbuzie AJ, Burke Johnson R. The “validity” issue in mixed research. Research in the Schools. 2006; 13 :48–63. [ Google Scholar ]
• Roth LM. Selling women short: Gender and money on Wall Street. Princeton: Princeton University Press; 2006. [ Google Scholar ]
• Schoonenboom J. The multilevel mixed intact group analysis: a mixed method to seek, detect, describe and explain differences between intact groups. Journal of Mixed Methods Research. 2016; 10 :129–146. doi: 10.1177/1558689814536283. [ CrossRef ] [ Google Scholar ]
• Schoonenboom, Judith, R. Burke Johnson, and Dominik E. Froehlich. 2017, in press. Combining multiple purposes of mixing within a mixed methods research design. International Journal of Multiple Research Approaches .
• Teddlie CB, Tashakkori A. Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Los Angeles: Sage; 2009. [ Google Scholar ]
• Yanchar SC, Williams DD. Reconsidering the compatibility thesis and eclecticism: Five proposed guidelines for method use. Educational Researcher. 2006; 35 (9):3–12. doi: 10.3102/0013189X035009003. [ CrossRef ] [ Google Scholar ]
• Yin RK. Case study research: design and methods. 5. Los Angeles: SAGE; 2013. [ Google Scholar ]

• Open access
• Published: 28 March 2024

Using the consolidated Framework for Implementation Research to integrate innovation recipients’ perspectives into the implementation of a digital version of the spinal cord injury health maintenance tool: a qualitative analysis

• John A Bourke 1 , 2 , 3 ,
• K. Anne Sinnott Jerram 1 , 2 ,
• Mohit Arora 1 , 2 ,
• Ashley Craig 1 , 2 &
• James W Middleton 1 , 2 , 4 , 5  

BMC Health Services Research volume  24 , Article number:  390 ( 2024 ) Cite this article

69 Accesses

Metrics details

Despite advances in managing secondary health complications after spinal cord injury (SCI), challenges remain in developing targeted community health strategies. In response, the SCI Health Maintenance Tool (SCI-HMT) was developed between 2018 and 2023 in NSW, Australia to support people with SCI and their general practitioners (GPs) to promote better community self-management. Successful implementation of innovations such as the SCI-HMT are determined by a range of contextual factors, including the perspectives of the innovation recipients for whom the innovation is intended to benefit, who are rarely included in the implementation process. During the digitizing of the booklet version of the SCI-HMT into a website and App, we used the Consolidated Framework for Implementation Research (CFIR) as a tool to guide collection and analysis of qualitative data from a range of innovation recipients to promote equity and to inform actionable findings designed to improve the implementation of the SCI-HMT.

Data from twenty-three innovation recipients in the development phase of the SCI-HMT were coded to the five CFIR domains to inform a semi-structured interview guide. This interview guide was used to prospectively explore the barriers and facilitators to planned implementation of the digital SCI-HMT with six health professionals and four people with SCI. A team including researchers and innovation recipients then interpreted these data to produce a reflective statement matched to each domain. Each reflective statement prefaced an actionable finding, defined as alterations that can be made to a program to improve its adoption into practice.

Five reflective statements synthesizing all participant data and linked to an actionable finding to improve the implementation plan were created. Using the CFIR to guide our research emphasized how partnership is the key theme connecting all implementation facilitators, for example ensuring that the tone, scope, content and presentation of the SCI-HMT balanced the needs of innovation recipients alongside the provision of evidence-based clinical information.

Conclusions

Understanding recipient perspectives is an essential contextual factor to consider when developing implementation strategies for healthcare innovations. The revised CFIR provided an effective, systematic method to understand, integrate and value recipient perspectives in the development of an implementation strategy for the SCI-HMT.

Trial registration

Peer Review reports

Injury to the spinal cord can occur through traumatic causes (e.g., falls or motor vehicle accidents) or from non-traumatic disease or disorder (e.g., tumours or infections) [ 1 ]. The onset of a spinal cord injury (SCI) is often sudden, yet the consequences are lifelong. The impact of a SCI is devastating, with effects on sensory and motor function, bladder and bowel function, sexual function, level of independence, community participation and quality of life [ 2 ]. In order to maintain good health, wellbeing and productivity in society, people with SCI must develop self-management skills and behaviours to manage their newly acquired chronic health condition [ 3 ]. Given the increasing emphasis on primary health care and community management of chronic health conditions, like SCI, there is a growing responsibility on all parties to promote good health practices and minimize the risks of common health complications in their communities.

To address this need, the Spinal Cord Injury Health Maintenance Tool (SCI-HMT) was co-designed between 2018 and 2023 with people living with SCI and their General Practitioners (GPs) in NSW, Australia [ 4 ] The aim of the SCI-HMT is to support self-management of the most common and arguably avoidable potentially life-threatening complications associated with SCI, such as mental health crises, autonomic dysreflexia, kidney infections and pressure injuries. The SCI-HMT provides comprehensible information with resources about the six highest priority health areas related to SCI (as indicated by people with SCI and GPs) and was developed over two phases. Phase 1 focused on developing a booklet version and Phase 2 focused on digitizing this content into a website and smartphone app [ 4 , 5 ].

Enabling the successful implementation of evidence-based innovations such as the SCI-HMT is inevitably influenced by contextual factors: those dynamic and diverse array of forces within real-world settings working for or against implementation efforts [ 6 ]. Contextual factors often include background environmental elements in which an intervention is situated, for example (but not limited to) demographics, clinical environments, organisational culture, legislation, and cultural norms [ 7 ]. Understanding the wider context is necessary to identify and potentially mitigate various challenges to the successful implementation of those innovations. Such work is the focus of determinant frameworks, which focus on categorising or classing groups of contextual determinants that are thought to predict or demonstrate an effect on implementation effectiveness to better understand factors that might influence implementation outcomes [ 8 ].

One of the most highly cited determinant frameworks is the Consolidated Framework for Implementation Research (CFIR) [ 9 ], which is often posited as an ideal framework for pre-implementation preparation. Originally published in 2009, the CFIR has recently been subject to an update by its original authors, which included a literature review, survey of users, and the creation of an outcome addendum [ 10 , 11 ]. A key contribution from this revision was the need for a greater focus on the place of innovation recipients, defined as the constituency for whom the innovation is being designed to benefit; for example, patients receiving treatment, students receiving a learning activity. Traditionally, innovation recipients are rarely positioned as key decision-makers or innovation implementers [ 8 ], and as a consequence, have not often been included in the application of research using frameworks, such as the CFIR [ 11 ].

Such power imbalances within the intersection of healthcare and research, particularly between those receiving and delivering such services and those designing such services, have been widely reported [ 12 , 13 ]. There are concerted efforts within health service development, health research and health research funding, to rectify this power imbalance [ 14 , 15 ]. Importantly, such efforts to promote increased equitable population impact are now being explicitly discussed within the implementation science literature. For example, Damschroder et al. [ 11 ] has recently argued for researchers to use the CFIR to collect data from innovation recipients, and that, ultimately, “equitable population impact is only possible when recipients are integrally involved in implementation and all key constituencies share power and make decisions together” (p. 7). Indeed, increased equity between key constituencies and partnering with innovation recipients promotes the likelihood of sustainable adoption of an innovation [ 4 , 12 , 14 ].

There is a paucity of work using the updated CFIR to include and understand innovation recipients’ perspectives. To address this gap, this paper reports on a process of using the CFIR to guide the collection of qualitative data from a range of innovation recipients within a wider co-design mixed methods study examining the development and implementation of SCI-HMT. The innovation recipients in our research are people living with SCI and GPs. Guided by the CFIR domains (shown in the supplementary material), we used reflexive thematic analysis [ 16 ]to summarize data into reflective summaries, which served to inform actionable findings designed to improve implementation of the SCI-HMT.

The procedure for this research is multi-stepped and is summarized in Fig.  1 . First, we mapped retrospective qualitative data collected during the development of the SCI-HMT [ 4 ] against the five domains of the CFIR in order to create a semi-structured interview guide (Step 1). Then, we used this interview guide to collect prospective data from health professionals and people with SCI during the development of the digital version of the SCI-HMT (Step 2) to identify implementation barriers and facilitators. This enabled us to interpret a reflective summary statement for each CFIR domain. Lastly, we developed an actionable finding for each domain summary. The first (RESP/18/212) and second phase (2019/ETH13961) of the project received ethical approval from The Northern Sydney Local Health District Human Research Ethics Committee. The reporting of this study was conducted in line with the consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines [ 17 ]. All methods were performed in accordance with the relevant guidelines and regulations.

Procedure of synthesising datasets to inform reflective statements and actionable findings. a Two health professionals had a SCI (one being JAB); b Two co-design researchers had a SCI (one being JAB)

Step one: retrospective data collection and analysis

We began by retrospectively analyzing the data set (interview and focus group transcripts) from the previously reported qualitative study from the development phase of the SCI-HMT [ 4 ]. This analysis was undertaken by two team members (KASJ and MA). KASJ has a background in co-design research. Transcript data were uploaded into NVivo software (Version 12: QSR International Pty Ltd) and a directed content analysis approach [ 18 ] was applied to analyze categorized data a priori according to the original 2009 CFIR domains (intervention characteristics, outer setting, inner setting, characteristics of individuals, and process of implementation) described by Damschroder et al. [ 9 ]. This categorized data were summarized and informed the specific questions of a semi-structured interview guide. The final output of step one was an interview guide with context-specific questions arranged according to the CFIR domains (see supplementary file 1). The interview was tested with two people with SCI and one health professional.

Step two: prospective data collection and analysis

In the second step, semi-structured interviews were conducted by KASJ (with MA as observer) with consenting healthcare professionals who had previously contributed to the development of the SCI-HMT. Healthcare professionals included GPs, Nurse Consultants, Specialist Physiotherapists, along with Health Researchers (one being JAB). In addition, a focus group was conducted with consenting individuals with SCI who had contributed to the SCI-HMT design and development phase. The interview schedule designed in step one above guided data collection in all interviews and the focus group.

The focus group and interviews were conducted online, audio recorded, transcribed verbatim and uploaded to NVivo software (Version 12: QSR International Pty Ltd). All data were subject to reflexive, inductive and deductive thematic analysis [ 16 , 19 ] to better understand participants’ perspectives regarding the potential implementation of the SCI-HMT. First, one team member (KASJ) read transcripts and began a deductive analysis whereby data were organized into CFIR domains-specific dataset. Second, KASJ and JAB analyzed this domain-specific dataset to inductively interpret a reflective statement which served to summarise all participant responses to each domain. The final output of step two was a reflective summary statement for each CFIR domain.

Step three: data synthesis

In the third step we aimed to co-create an actionable finding (defined as tangible alteration that can be made to a program, in this case the SCI-HMT [ 20 ]) based on each domain-specific reflective statement. To achieve this, three codesign researchers (KAS and JAB with one person with SCI from Step 2 (deidentified)) focused on operationalising each reflective statement into a recommended modification for the digital version of the SCI-HMT. This was an iterative process guided by the specific CFIR domain and construct definitions, which we deemed salient and relevant to each reflective statement (see Table  2 for example). Data synthesis involved line by line analysis, group discussion, and repeated refinement of actionable findings. A draft synthesis was shared with SCI-HMT developers (JWM and MA) and refinement continued until consensus was agreed on. The final outputs of step three were an actionable finding related to each reflective statement for each CFIR domain.

The characteristics of both the retrospective and prospective study participants are shown in Table  1 . The retrospective data included data from a total of 23 people: 19 people with SCI and four GPs. Of the 19 people with SCI, 12 participated in semi-structured interviews, seven participated in the first focus group, and four returned to the second focus group. In step 2, four people with SCI participated in a focus group and six healthcare professionals participated in one-on-one semi-structured interviews. Two of the healthcare professionals (a GP and a registrar) had lived experience of SCI, as did one researcher (JAB). All interviews and focus groups were conducted either online or in-person and ranged in length between 60 and 120 min.

In our overall synthesis, we actively interpreted five reflective statements based on the updated CFIR domain and construct definitions by Damschroder et al. [ 11 ]. Table  2 provides a summary of how we linked the updated CFIR domain and construct definitions to the reflective statements. We demonstrate this process of co-creation below, including illustrative quotes from participants. Importantly, we guide readers to the actionable findings related to each reflective statement in Table  2 . Each actionable statement represents an alteration that can be made to a program to improve its adoption into practice.

Participants acknowledged that self-management is a major undertaking and very demanding, as one person with SCI said, “ we need to be informed without being terrified and overwhelmed”. Participants felt the HMT could indeed be adapted, tailored, refined, or reinvented to meet local needs. For example, another person with SCI remarked:

“Education needs to be from the get-go but in bite sized pieces from all quarters when readiness is most apparent… at all time points , [not just as a] a newbie tool or for people with [long-term impairment] ” (person with SCI_02).

Therefore, the SCI-HMT had to balance complexity of content while still being accessible and engaging, and required input from both experts in the field and those with lived experience of SCI, for example, a clinical nurse specialist suggested:

“it’s essential [the SCI-HMT] is written by experts in the field as well as with collaboration with people who have had a, you know, the lived experience of SCI” (healthcare professional_03).

Furthermore, the points of contact with healthcare for a person with SCI can be challenging to navigate and the SCI-HMT has the potential to facilitate a smoother engagement process and improve communication between people with SCI and healthcare services. As a GP suggested:

“we need a tool like this to link to that pathway model in primary health care , [the SCI-HMT] it’s a great tool, something that everyone can read and everyone’s reading the same thing” (healthcare professional_05).

Participants highlighted that the ability of the SCI-HMT to facilitate effective communication was very much dependent on the delivery format. The idea of digitizing the SCI-HMT garnered equal support from people with SCI and health care professionals, with one participant with SCI deeming it to be “ essential” ( person with SCI_01) and a health professional suggesting a “digitalized version will be an advantage for most people” (healthcare professional_02).

Outer setting

There was strong interest expressed by both people with SCI and healthcare professionals in using the SCI-HMT. The fundamental premise was that knowledge is power and the SCI-HMT would have strong utility in post-acute rehabilitation services, as well as primary care. As a person with SCI said,

“ we need to leave the [spinal unit] to return to the community with sufficient knowledge, and to know the value of that knowledge and then need to ensure primary healthcare provider [s] are best informed” (person with SCI_04).

The value of the SCI-HMT in facilitating clear and effective communication and shared decision-making between healthcare professionals and people with SCI was also highlighted, as shown by the remarks of an acute nurse specialist:

“I think this tool is really helpful for the consumer and the GP to work together to prioritize particular tests that a patient might need and what the regularity of that is” (healthcare professional_03).

Engaging with SCI peer support networks to promote the SCI-HMT was considered crucial, as one person with SCI emphasized when asked how the SCI-HMT might be best executed in the community, “…peers, peers and peers” (person with SCI_01). Furthermore, the layering of content made possible in the digitalized version will allow for the issue of approachability in terms of readiness for change, as another person with SCI said:

“[putting content into a digital format] is essential and required and there is a need to put summarized content in an App with links to further web-based information… it’s not likely to be accessed otherwise” (person with SCI_02).

Inner setting

Participants acknowledged that self-management of health and well-being is substantial and demanding. It was suggested that the scope, tone, and complexity of the SCI-HMT, while necessary, could potentially be resisted by people with SCI if they felt overwhelmed, as one person with SCI described:

“a manual that is really long and wordy, like, it’s [a] health metric… they maybe lack the health literacy to, to consume the content then yes, it would impede their readiness for [self-management]” (person with SCI_02).

Having support from their GPs was considered essential, and the HMT could enable GP’s, who are under time pressure, to provide more effective health and advice to their patients, as one GP said:

“We GP’s are time poor, if you realize then when you’re time poor you look quickly to say oh this is a patient tool - how can I best use this?” (healthcare professional_05).

Furthermore, health professional skills may be best used with the synthesis of self-reported symptoms, behaviors, or observations. A particular strength of a digitized version would be its ability to facilitate more streamlined communication between a person with SCI and their primary healthcare providers developing healthcare plans, as an acute nurse specialist reflected, “ I think that a digitalized version is essential with links to primary healthcare plans” (healthcare professional_03).

Efficient communication with thorough assessment is essential to ensure serious health issues are not missed, as findings reinforce that the SCI-HMT is an educational tool, not a replacement for healthcare services, as a clinical nurse specialist commented, “ remember, things will go wrong– people end up very sick and in acute care “ (healthcare professional_02).

The SCI-HMT has the potential to provide a pathway to a ‘hope for better than now’ , a hope to ‘remain well’ and a hope to ‘be happy’ , as the informant with SCI (04) declared, “self-management is a long game, if you’re keeping well, you’ve got that possibility of a good life… of happiness”. Participants with SCI felt the tool needed to be genuine and

“acknowledge the huge amount of adjustment required, recognizing that dealing with SCI issues is required to survive and live a good life” (person with SCI_04).

However, there is a risk that an individual is completely overwhelmed by the scale of the SCI-HMT content and the requirement for lifelong vigilance. Careful attention and planning were paid to layering the information accordingly to support self-management as a ‘long game’, which one person with SCI reflected in following:

“the first 2–3 year [period] is probably the toughest to get your head around the learning stuff, because you’ve got to a stage where you’re levelling out, and you’ve kind of made these promises to yourself and then you realize that there’s no quick fix” (person with SCI_01).

It was decided that this could be achieved by providing concrete examples and anecdotes from people with SCI illustrating that a meaningful, healthy life is possible, and that good health is the bedrock of a good life with SCI.

There was universal agreement that the SCI-HMT is aspirational and that it has the potential to improve knowledge and understanding for people with SCI, their families, community workers/carers and primary healthcare professionals, as a GP remarked:

“[different groups] could just read it and realize, ‘Ahh, OK that’s what that means… when you’re doing catheters. That’s what you mean when you’re talking about bladder and bowel function or skin care” (healthcare professional_04).

Despite the SCI-HMT providing an abundance of information and resources to support self-management, participants identified four gaps: (i) the priority issue of sexuality, including pleasure and identity, as one person with SCI remarked:

“ sexuality is one of the biggest issues that people with SCI often might not speak about that often cause you know it’s awkward for them. So yeah, I think that’s a that’s a serious issue” (person with SCI_03).

(ii) consideration of the taboo nature of bladder and bowel topics for indigenous people, (iii) urgent need to ensure links for SCI-HMT care plans are compatible with patient management systems, and (iv) exercise and leisure as a standalone topic taking account of effects of physical activity, including impact on mental health and wellbeing but more especially for fun.

To ensure longevity of the SCI-HMT, maintaining a partnership between people with SCI, SCI community groups and both primary and tertiary health services is required for liaison with the relevant professional bodies, care agencies, funders, policy makers and tertiary care settings to ensure ongoing education and promotion of SCI-HMT is maintained. For example, delivery of ongoing training of healthcare professionals to both increase the knowledge base of primary healthcare providers in relation to SCI, and to promote use of the tools and resources through health communities. As a community nurse specialist suggested:

“ improving knowledge in the health community… would require digital links to clinical/health management platforms” (healthcare professional_02).

In a similar vein, a GP suggested:

“ our common GP body would have continuing education requirements… especially if it’s online, in particular for the rural, rural doctors who you know, might find it hard to get into the city” (healthcare professional_04).

The successful implementation of evidence-based innovations into practice is dependent on a wide array of dynamic and active contextual factors, including the perspectives of the recipients who are destined to use such innovations. Indeed, the recently updated CFIR has called for innovation recipient perspectives to be a priority when considering contextual factors [ 10 , 11 ]. Understanding and including the perspectives of those the innovation is being designed to benefit can promote increased equity and validation of recipient populations, and potentially increase the adoption and sustainability of innovations.

In this paper, we have presented research using the recently updated CFIR to guide the collection of innovation recipients’ perspectives (including people with SCI and GPs working in the community) regarding the potential implementation barriers and facilitators of the digital version of the SCI-HMT. Collected data were synthesized to inform actionable findings– tangible ways in which the SCI-HMT could be modified according of the domains of the CFIR (e.g., see Keith et al. [ 20 ]). It is important to note that we conducted this research using the original domains of the CFIR [ 9 ] prior to Damschroder et al. publishing the updated CFIR [ 11 ]. However, in our analysis we were able to align our findings to the revised CFIR domains and constructs, as Damschroder [ 11 ] suggests, constructs can “be mapped back to the original CFIR to ensure longitudinal consistency” (p. 13).

One of the most poignant findings from our analyses was the need to ensure the content of the SCI-HMT balanced scientific evidence and clinical expertise with lived experience knowledge. This balance of clinical and experiential knowledge demonstrated genuine regard for lived experience knowledge, and created a more accessible, engaging, useable platform. For example, in the innovation and individual domains, the need to include lived experience quotes was immediately apparent once the perspective of people with SCI was included. It was highlighted that while the SCI-HMT will prove useful to many parties at various stages along the continuum of care following onset of SCI, there will be those individuals that are overwhelmed by the scale of the content. That said, the layering of information facilitated by the digitalized version is intended to provide an ease of navigation through the SCI-HMT and enable a far greater sense of control over personal health and wellbeing. Further, despite concerns regarding e-literacy the digitalized version of the SCI-HMT is seen as imperative for accessibility given the wide geographic diversity and recent COVID pandemic [ 21 ]. While there will be people who are challenged by the technology, the universally acceptable use of the internet is seen as less of a barrier than printed material.

The concept of partnership was also apparent within the data analysis focusing on the outer and inner setting domains. In the outer setting domain, our findings emphasized the importance of engaging with SCI community groups, as well as primary and tertiary care providers to maximize uptake at all points in time from the phase of subacute rehabilitation onwards. While the SCI-HMT is intended for use across the continuum of care from post-acute rehabilitation onwards, it may be that certain modules are more relevant at different times, and could serve as key resources during the hand over between acute care, inpatient rehabilitation and community reintegration.

Likewise, findings regarding the inner setting highlighted the necessity of a productive partnership between GPs and individuals with SCI to address the substantial demands of long-term self-management of health and well-being following SCI. Indeed, support is crucial, especially when self-management is the focus. This is particularly so in individuals living with complex disability following survival after illness or injury [ 22 ], where health literacy has been found to be a primary determinant of successful health and wellbeing outcomes [ 23 ]. For people with SCI, this tool potentially holds the most appeal when an individual is ready and has strong partnerships and supportive communication. This can enable potential red flags to be recognized earlier allowing timely intervention to avert health crises, promoting individual well-being, and reducing unnecessary demands on health services.

While the SCI-HMT is an educational tool and not meant to replace health services, findings suggest the current structure would lead nicely to having the conversation with a range of likely support people, including SCI peers, friends and family, GP, community nurses, carers or via on-line support services. The findings within the process domain underscored the importance of ongoing partnership between innovation implementers and a broad array of innovation recipients (e.g., individuals with SCI, healthcare professionals, family, funding agencies and policy-makers). This emphasis on partnership also addresses recent discussions regarding equity and the CFIR. For example, Damschroder et al. [ 11 ] suggests that innovation recipients are too often not included in the CFIR process, as the CFIR is primarily seen as a tool intended “to collect data from individuals who have power and/or influence over implementation outcomes” (p. 5).

Finally, we feel that our inclusion of innovation recipients’ perspectives presented in this article begins to address the notion of equity in implementation, whereby the inclusion of recipient perspectives in research using the CFIR both validates, and increases, the likelihood of sustainable adoption of evidence-based innovations, such as the SCI-HMT. We have used the CFIR in a pragmatic way with an emphasis on meaningful engagement between the innovation recipients and the research team, heeding the call from Damschroder et al. [ 11 ], who recently argued for researchers to use the CFIR to collect data from innovation recipients. Adopting this approach enabled us to give voice to innovation recipient perspectives and subsequently ensure that the tone, scope, content and presentation of the SCI-HMT balanced the needs of innovation recipients alongside the provision of evidence-based clinical information.

Our research is not without limitations. While our study was successful in identifying a number of potential barriers and facilitators to the implementation of the SCI-HMT, we did not test any implementation strategies to impact determinants, mechanisms, or outcomes. This will be the focus of future research on this project, which will investigate the impact of implementation strategies on outcomes. Focus will be given to the context-mechanism configurations which give rise to particular outcomes for different groups in certain circumstances [ 7 , 24 ]. A second potential concern is the relatively small sample size of participants that may not allow for saturation and generalizability of the findings. However, both the significant impact of secondary health complications for people with SCI and the desire for a health maintenance tool have been established in Australia [ 2 , 4 ]. The aim our study reported in this article was to achieve context-specific knowledge of a small sample that shares a particular mutual experience and represents a perspective, rather than a population [ 25 , 26 ]. We feel our findings can stimulate discussion and debate regarding participant-informed approaches to implementation of the SCI-HMT, which can then be subject to larger-sample studies to determine their generalisability, that is, their external validity. Notably, future research could examine the interaction between certain demographic differences (e.g., gender) of people with SCI and potential barriers and facilitators to the implementation of the SCI-HMT. Future research could also include the perspectives of other allied health professionals working in the community, such as occupational therapists. Lastly, while our research gave significant priority to recipient viewpoints, research in this space would benefit for ensuring innovation recipients are engaged as genuine partners throughout the entire research process from conceptualization to implementation.

Employing the CFIR provided an effective, systematic method for identifying recipient perspectives regarding the implementation of a digital health maintenance tool for people living with SCI. Findings emphasized the need to balance clinical and lived experience perspectives when designing an implementation strategy and facilitating strong partnerships with necessary stakeholders to maximise the uptake of SCI-HMT into practice. Ongoing testing will monitor the uptake and implementation of this innovation, specifically focusing on how the SCI-HMT works for different users, in different contexts, at different stages and times of the rehabilitation journey.

Data availability

The datasets supporting the conclusions of this article are available available upon request and with permission gained from the project Steering Committee.

Abbreviations

spinal cord injury

HMT-Spinal Cord Injury Health Maintenance Tool

Consolidated Framework for Implementation Research

Kirshblum S, Vernon WL. Spinal Cord Medicine, Third Edition. New York: Springer Publishing Company; 2018.

Middleton JW, Arora M, Kifley A, Clark J, Borg SJ, Tran Y, et al. Australian arm of the International spinal cord Injury (Aus-InSCI) Community Survey: 2. Understanding the lived experience in people with spinal cord injury. Spinal Cord. 2022;60(12):1069–79.

Article   PubMed   PubMed Central   Google Scholar  

Craig A, Nicholson Perry K, Guest R, Tran Y, Middleton J. Adjustment following chronic spinal cord injury: determining factors that contribute to social participation. Br J Health Psychol. 2015;20(4):807–23.

Article   PubMed   Google Scholar  

Middleton JW, Arora M, Jerram KAS, Bourke J, McCormick M, O’Leary D, et al. Co-design of the Spinal Cord Injury Health Maintenance Tool to support Self-Management: a mixed-methods Approach. Top Spinal Cord Injury Rehabilitation. 2024;30(1):59–73.

Article   Google Scholar  

Middleton JW, Arora M, McCormick M, O’Leary D. Health maintenance Tool: how to stay healthy and well with a spinal cord injury. A tool for consumers by consumers. 1st ed. Sydney, NSW Australia: Royal Rehab and The University of Sydney; 2020.

Google Scholar  

Nilsen P, Bernhardsson S. Context matters in implementation science: a scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Serv Res. 2019;19(1):189.

Jagosh J. Realist synthesis for Public Health: building an Ontologically Deep understanding of how Programs Work, for whom, and in which contexts. Annu Rev Public Health. 2019;40(1):361–72.

Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):53.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.

Damschroder LJ, Reardon CM, Opra Widerquist MA, Lowery JC. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR outcomes Addendum. Implement Sci. 2022;17(1):7.

Damschroder LJ, Reardon CM, Widerquist MAO, Lowery JC. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.

Plamondon K, Ndumbe-Eyoh S, Shahram S. 2.2 Equity, Power, and Transformative Research Coproduction. Research Co-Production in Healthcare2022. p. 34–53.

Verville L, Cancelliere C, Connell G, Lee J, Munce S, Mior S, et al. Exploring clinicians’ experiences and perceptions of end-user roles in knowledge development: a qualitative study. BMC Health Serv Res. 2021;21(1):926.

Gainforth HL, Hoekstra F, McKay R, McBride CB, Sweet SN, Martin Ginis KA, et al. Integrated Knowledge Translation Guiding principles for conducting and Disseminating Spinal Cord Injury Research in Partnership. Arch Phys Med Rehabil. 2021;102(4):656–63.

Langley J, Knowles SE, Ward V. Conducting a Research Coproduction Project. Research Co-Production in Healthcare2022. p. 112– 28.

Braun V, Clarke V. One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology. 2020:1–25.

Tong A, Sainsbury p, Craig J. Consolidated criteria for reporting qulaitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Bengtsson M. How to plan and perform a qualitative study using content analysis. NursingPlus Open. 2016;2:8–14.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77–101.

Keith RE, Crosson JC, O’Malley AS, Cromp D, Taylor EF. Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implement Science: IS. 2017;12(1):15.

Choukou M-A, Sanchez-Ramirez DC, Pol M, Uddin M, Monnin C, Syed-Abdul S. COVID-19 infodemic and digital health literacy in vulnerable populations: a scoping review. Digit HEALTH. 2022;8:20552076221076927.

PubMed   PubMed Central   Google Scholar  

Daniels N. Just Health: Meeting Health needs fairly. Cambridge University Press; 2007. p. 397.

Parker SM, Stocks N, Nutbeam D, Thomas L, Denney-Wilson E, Zwar N, et al. Preventing chronic disease in patients with low health literacy using eHealth and teamwork in primary healthcare: protocol for a cluster randomised controlled trial. BMJ Open. 2018;8(6):e023239–e.

Salter KL, Kothari A. Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implement Sci. 2014;9(1):115.

Sebele-Mpofu FY. The Sampling Conundrum in qualitative research: can Saturation help alleviate the controversy and alleged subjectivity in Sampling? Int’l J Soc Sci Stud. 2021;9:11.

Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies: guided by Information Power. Qual Health Res. 2015;26(13):1753–60.

Download references

Acknowledgements

Authors of this study would like to thank all the consumers with SCI and healthcare professionals for their invaluable contribution to this project. Their participation and insights have been instrumental in shaping the development of the SCI-HMT. The team also acknowledges the support and guidance provided by the members of the Project Steering Committee, as well as the partner organisations, including NSW Agency for Clinical Innovation, and icare NSW. Author would also like to acknowledge the informant group with lived experience, whose perspectives have enriched our understanding and informed the development of SCI-HMT.

The SCI Wellness project was a collaborative project between John Walsh Centre for Rehabilitation Research at The University of Sydney and Royal Rehab. Both organizations provided in-kind support to the project. Additionally, the University of Sydney and Royal Rehab received research funding from Insurance and Care NSW (icare NSW) to undertake the SCI Wellness Project. icare NSW do not take direct responsibility for any of the following: study design, data collection, drafting of the manuscript, or decision to publish.

Author information

Authors and affiliations.

John Walsh Centre for Rehabilitation Research, Northern Sydney Local Health District, St Leonards, NSW, Australia

John A Bourke, K. Anne Sinnott Jerram, Mohit Arora, Ashley Craig & James W Middleton

The Kolling Institute, Faculty of Medicine and Health, The University of Sydney, Sydney, NSW, Australia

Burwood Academy Trust, Burwood Hospital, Christchurch, New Zealand

John A Bourke

Royal Rehab, Ryde, NSW, Australia

James W Middleton

State Spinal Cord Injury Service, NSW Agency for Clinical Innovation, St Leonards, NSW, Australia

You can also search for this author in PubMed   Google Scholar

Contributions

Project conceptualization: KASJ, MA, JWM; project methodology: JWM, MA, KASJ, JAB; data collection: KASJ and MA; data analysis: KASJ, JAB, MA, JWM; writing—original draft preparation: JAB; writing—review and editing: JAB, KASJ, JWM, MA, AC; funding acquisition: JWM, MA. All authors contributed to the revision of the paper and approved the final submitted version.

Corresponding author

Correspondence to John A Bourke .

Ethics declarations

Ethics approval and consent to participate.

The first (RESP/18/212) and second phase (2019/ETH13961) of the project received ethical approval from The Northern Sydney Local Health District Human Research Ethics Committee. All participants provided informed, written consent. All data were to be retained for 7 years (23rd May 2030).

Consent for publication

Not applicable.

Competing interests

MA part salary (from Dec 2018 to Dec 2023), KASJ part salary (July 2021 to Dec 2023) and JAB part salary (Jan 2022 to Aug 2022) was paid from the grant monies. Other authors declare no conflicts of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Bourke, J.A., Jerram, K.A.S., Arora, M. et al. Using the consolidated Framework for Implementation Research to integrate innovation recipients’ perspectives into the implementation of a digital version of the spinal cord injury health maintenance tool: a qualitative analysis. BMC Health Serv Res 24 , 390 (2024). https://doi.org/10.1186/s12913-024-10847-x

Download citation

Received : 14 August 2023

Accepted : 11 March 2024

Published : 28 March 2024

DOI : https://doi.org/10.1186/s12913-024-10847-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Spinal Cord injury
• Self-management
• Innovation recipients
• Secondary health conditions
• Primary health care
• Evidence-based innovations
• Actionable findings
• Consolidated Framework for implementation research

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Design and Development Research

• First Online: 01 January 2013

Cite this chapter

• Rita C. Richey 5 &
• James D. Klein 6  

32k Accesses

27 Citations

This chapter focuses on design and development research, a type of inquiry unique to the instructional design and technology field dedicated to the creation of new knowledge and the validation of existing practice. We first define this kind of research and provide an overview of its two main categories—research on products and tools and research on design and development models. Then, we concentrate on recent design and development research (DDR) by describing 11 studies published in the literature. The five product and tool studies reviewed include research on comprehensive development projects, studies of particular design and development phases, and research on tool development and use. The six model studies reviewed include research leading to new or enhanced ID models, model validation and model use research. Finally, we summarize this new work in terms of the problems it addresses, the settings and participants examined, the research methodologies employed used, and the role evaluation plays in these studies.

• Design and development research
• Instructional and non-instructional products
• Design and development tools
• Instructional design models

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

*Cifuentes, L., Sharp, A., Bulu, S., Benz, M., & Stough, L. M. (2010). Developing a Web 2.0-based system with user-authored content for community use and teacher education. Educational Technology Research and Development, 58 (4), 377–398.

Google Scholar  

Coetzee, J. S., & Smart, A. (2012). Rapid implementation of e-learning using a technology design model. In N. A. Alias & S. Hashim (Eds.), Instructional technology research, design, and development: Lessons from the field (pp. 219–237). Hershey, PA: IGI Global.

Cowell, D. (2001). Needs assessment activities and techniques of instructional designers: A qualitative study (Doctoral dissertation, Wayne State University, 2000). Dissertation Abstracts International A, 61 (910), 3873.

*Crossman, D. C. (2010). Gilbert’s Behavioral Engineering Model: Contemporary support for an established theory. Performance Improvement Quarterly, 23 (1), 31–52.

Definition and Terminology Committee of the Association for Educational Communications & Technology. (2007). Definition. In A. Januszewski & M. Molenda (Eds.), Educational technology: A definition with commentary (pp. 1–14). New York, NY: Routledge.

Dick, W., Carey, L., & Carey, J. O. (2009). The systematic design of instruction (7th ed.). Upper Saddle River, NJ: Merrill.

Driscoll, M. P. (1984). Paradigms for research in instructional systems. Journal of Instructional Development, 7 (4), 2–5.

Article   Google Scholar  

Fischer, K. M., Savenye, W. C., & Sullivan, H. J. (2002). Formative evaluation of computer-based training for a university financial system. Performance Improvement Quarterly, 15 (1), 11–24.

Francom, G., Bybee, D., Wolfersberger, M., & Merrill, M. D. (2009). Biology 100: A task-centered, peer-interactive redesign. TechTrends, 53 (4), 35–42.

Gilbert, T. (1978). Human competence: Engineering worthy performance . New York, NY: McGraw-Hill.

Hart, S. M. (2008). The design decisions of teachers during technology integration . (Unpublished doctoral dissertation). Wayne State University, Detroit, MI.

Hillsman, T. L., & Kuptritz, V. W. (2010). Another look at the relative impact of workplace design on training transfer for supervisory communication skills. Performance Improvement Quarterly, 23 (3), 107–130.

*Hung, W-C., Smith, T. J., Harris, M. S., & Lockard, J. (2010). Development research of a teachers’ educational performance support system: The practices of design, development, and evaluation. Educational Technology Research and Development, 58 (1), 61–80.

Jones, T. S., & Richey, R. C. (2000). Rapid prototyping in action: A developmental study. Educational Technology Research and Development, 48 (2), 63–80.

Klein, J. D. (1997). ETR&D—Development: An analysis of content and survey of future direction. Educational Technology Research and Development, 45 (3), 57–62.

Klein, J. D., Martin, F., Tutty, J., & Su, Y. (2005). Teaching research to instructional design & technology students. Educational Technology, 45 (4), 29–33.

Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43–59.

Mooij, T. (2002). Designing a digital instructional management system to optimize early education. Educational Technology Research and Development, 50 (4), 11–23.

Nieveen, N., & van den Akker, J. (1999). Exploring the potential of a computer tool for instructional developers. Educational Technology Research and Development, 47 (3), 77–98.

Plass, J. L., & Salisbusry, M. W. (2002). A living-systems design model for web-based knowledge management systems. Educational Technology Research and Development, 50 (1), 35–57.

Reigeluth, C. M., & Frick, T. W. (1999). Formative research: A methodology for creating and improving design theories. In C. M. Reigeluth (Ed.), Instructional design theories and models, volume II: A new paradigm of instructional theory (pp. 633–651). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Reiser, R. A. (2012). What field did you say you were in? Defining and naming our field. In R. A. Reiser & J. V. Dempsey (Eds.), Trends and issues in instructional design and technology (3rd ed., pp. 1–7). Upper Saddle River, NJ: Merrill.

Richey, R. C. (1997). Research on instructional development. Educational Technology Research and Development, 45 (3), 91–100.

*Richey, R. C. (2005). Validating instructional design and development models. In J. M. Spector & D.A. Wiley (Eds.), Innovations in instructional technology: Essays in honor of M. David Merrill (pp. 171–185). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Richey, R. C., & Klein, J. D. (2005). Developmental research methods: Creating knowledge from instructional design and development practice. Journal of Computing in Higher Education, 16 (2), 23–38.

*Richey, R. C., & Klein, J. D. (2007). Design and development research: Methods, strategies and issues . Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Richey, R. C., & Klein, J. D. (2008). Research on design and development. In J. M. Spector, M. D. Merrill, J. van Merrienboer, & M. P. Driscoll (Eds.), Handbook of research for educational communications and technology (3rd ed., pp. 748–757). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

*Richey, R. C., Klein, J. D., & Nelson, W. (2004). Developmental research: Studies of instructional design and development. In D. Jonassen (Ed.), Handbook of research for educational communications and technology (2nd ed., pp. 1099–1130). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Richey, R. C., Klein, J. D., & Tracey, M. W. (2011). The instructional design knowledge base: Theory, research and practice . New York, NY: Routledge.

Richey, R. C., & Nelson, W. (1996). Developmental research. In D. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 1213–1245). New York, NY: Simon & Schuster.

Roszkowski, M. J., & Soven, M. (2010). Did you learning something useful today? An analysis of how perceived utility relates to perceived learning and their predictiveness of satisfaction with training. Performance Improvement Quarterly, 23 (2), 71–91.

Roytek, M. A. (2000). Contextual factors affecting the use of rapid ­prototyping within the design and development of instruction (Doctoral dissertation, Wayne State University, 1999). Dissertation Abstracts International A, 61 (01), 76.

Sahrir, M. (2012). Formative evaluation of an Arabic online vocabulary learning game prototype: Lessons from a Malaysian institute of higher learning experience. In N. Alias & S. Hashim (Eds.), Instructional technology research, design and development: Lessons from the field (pp. 357–368). Hershey, PA: IGI Global.

Sullivan, H., Ice, K., & Niedermeyer, F. (2000). Long-term instructional development: A 20-year ID and implementation project. Educational Technology Research and Development, 48 (4), 87–99.

Tessmer, M. (1993). Planning and conducting formative evaluation: Improving the quality of education and training . London: Kogan Page.

Tracey, M. S. (2009). Design and development research: A model validation case. Educational Technology Research and Development, 57 (4), 553–571.

Vallachia, S. W., Marker, A., & Taylor, K. (2010). But what do they really expect? Employer perceptions of the skills of entry-level instructional designers. Performance Improvement Quarterly, 22 (4), 33–51.

*van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, R. M. Branch, K. Gustafson, N. Nieveen & T. Plomp (Eds.), Design approaches and tools in education and training (pp. 1–14). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Visscher-Voerman, I., & Gustafson, K. L. (2004). Paradigms in the theory and practice of education and training design. Educational Technology Research and Development, 52 (2), 69–89.

Visser, L., Plomp, T., Armiault, R. J., & Kuiper, W. (2002). Motivating students at a distance: The case of an international audience. Educational Technology Research and Development, 50 (2), 94–110.

*Voss, D. R. (2008). The development of a model for non-verbal factors impacting the design of visual information. Unpublished doctoral dissertation, Wayne State University, Detroit, MI.

Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53 (4), 5–23.

Wang, Q., Nieveen, N., & van den Akker, J. (2007). Designing a computer support system for multimedia curriculum development in Shanghai. Educational Technology Research and Development, 55 (3), 275–295.

Download references

Author information

Authors and affiliations.

Wayne State University, Detroit, MI, USA

Rita C. Richey

Florida State University, Tallahassee, FL, USA

James D. Klein

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rita C. Richey .

Editor information

Editors and affiliations.

, Department of Learning Technologies, C, University of North Texas, North Elm 3940, Denton, 76207-7102, Texas, USA

J. Michael Spector

W. Sunset Blvd. 1812, St. George, 84770, Utah, USA

M. David Merrill

, Centr. Instructiepsychol.&-technologie, K.U. Leuven, Andreas Vesaliusstraat 2, Leuven, 3000, Belgium

Research Drive, Iacocca A109 111, Bethlehem, 18015, Pennsylvania, USA

M. J. Bishop

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media New York

About this chapter

Richey, R.C., Klein, J.D. (2014). Design and Development Research. In: Spector, J., Merrill, M., Elen, J., Bishop, M. (eds) Handbook of Research on Educational Communications and Technology. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-3185-5_12

Download citation

DOI : https://doi.org/10.1007/978-1-4614-3185-5_12

Published : 22 May 2013

Publisher Name : Springer, New York, NY

Print ISBN : 978-1-4614-3184-8

Online ISBN : 978-1-4614-3185-5

eBook Packages : Humanities, Social Sciences and Law Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 25 March 2024

Pervasive environmental chemicals impair oligodendrocyte development

• Erin F. Cohn 1 ,
• Benjamin L. L. Clayton   ORCID: orcid.org/0000-0001-9552-404X 1 ,
• Mayur Madhavan 1 ,
• Kristin A. Lee 1 ,
• Sara Yacoub 1 ,
• Yuriy Fedorov   ORCID: orcid.org/0000-0001-9513-8341 1 ,
• Marissa A. Scavuzzo 1 ,
• Katie Paul Friedman 2 ,
• Timothy J. Shafer 2 &
• Paul J. Tesar   ORCID: orcid.org/0000-0003-1532-3155 1  

Nature Neuroscience ( 2024 ) Cite this article

2419 Accesses

802 Altmetric

Metrics details

• Myelin biology and repair
• Oligodendrocyte

Exposure to environmental chemicals can impair neurodevelopment, and oligodendrocytes may be particularly vulnerable, as their development extends from gestation into adulthood. However, few environmental chemicals have been assessed for potential risks to oligodendrocytes. Here, using a high-throughput developmental screen in cultured cells, we identified environmental chemicals in two classes that disrupt oligodendrocyte development through distinct mechanisms. Quaternary compounds, ubiquitous in disinfecting agents and personal care products, were potently and selectively cytotoxic to developing oligodendrocytes, whereas organophosphate flame retardants, commonly found in household items such as furniture and electronics, prematurely arrested oligodendrocyte maturation. Chemicals from each class impaired oligodendrocyte development postnatally in mice and in a human 3D organoid model of prenatal cortical development. Analysis of epidemiological data showed that adverse neurodevelopmental outcomes were associated with childhood exposure to the top organophosphate flame retardant identified by our screen. This work identifies toxicological vulnerabilities for oligodendrocyte development and highlights the need for deeper scrutiny of these compounds’ impacts on human health.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 12 print issues and online access

195,33 € per year

only 16,28 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Formation of memory assemblies through the DNA-sensing TLR9 pathway

Vladimir Jovasevic, Elizabeth M. Wood, … Jelena Radulovic

The level of protein in the maternal murine diet modulates the facial appearance of the offspring via mTORC1 signaling

Meng Xie, Markéta Kaiser, … Andrei S. Chagin

A single-cell time-lapse of mouse prenatal development from gastrula to birth

Chengxiang Qiu, Beth K. Martin, … Jay Shendure

Data availability

Primary screening results are available in Supplementary Table 1 and will be included in a future public release of the EPA ToxCast database. RNA-seq datasets generated in this study have been deposited in Gene Expression Omnibus ( https://www.ncbi.nlm.nih.gov/geo ) under accession code GSE244500 . Data from the CDC’s NHANES utilized in this study is publicly available at https://wwwn.cdc.gov/Nchs/Nhanes . The mm10 genome utilized in RNA sequencing analysis is publicly available from GENCODE. Source data are provided with this paper.

Code availability

Epidemiological analyses were performed using publicly available packages and followed guidelines provided by NHANES.

Sanmarco, L. M. et al. Identification of environmental factors that promote intestinal inflammation. Nature 611 , 801–809 (2022).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Wheeler, M. A. et al. Environmental control of astrocyte pathogenic activities in CNS inflammation. Cell 176 , 581–596.e18 (2019).

Lidsky, T. I. & Schneider, J. S. Lead neurotoxicity in children: basic mechanisms and clinical correlates. Brain 126 , 5–19 (2003).

Article   PubMed   Google Scholar  

Grandjean, P. & Landrigan, P. J. Neurobehavioural effects of developmental toxicity. Lancet Neurol. 13 , 330–338 (2014).

Grandjean, P. & Landrigan, P. J. Developmental neurotoxicity of industrial chemicals. Lancet 368 , 2167–2178 (2006).

Article   CAS   PubMed   Google Scholar  

Landrigan, P. J. et al. Neuropsychological dysfunction in children with chronic low-level lead absorption. Lancet 1 , 708–712 (1975).

Jacobson, J. L. & Jacobson, S. W. Intellectual impairment in children exposed to polychlorinated biphenyls in utero. N. Engl. J. Med. 335 , 783–789 (1996).

Li, Q. et al. Prevalence of autism spectrum disorder among children and adolescents in the United States from 2019 to 2020. JAMA Pediatr. 176 , 943–945 (2022).

Article   PubMed   PubMed Central   Google Scholar  

Chung, W. et al. Trends in the prevalence and incidence of attention-deficit/hyperactivity disorder among adults and children of different racial and ethnic groups. JAMA Netw. Open 2 , e1914344 (2019).

Zhou, T. et al. A hPSC-based platform to discover gene-environment interactions that impact human beta-cell and dopamine neuron survival. Nat. Commun. 9 , 4815 (2018).

Caporale, N. et al. From cohorts to molecules: adverse impacts of endocrine disrupting mixtures. Science 375 , eabe8244 (2022).

Bercury, K. K. & Macklin, W. B. Dynamics and mechanisms of CNS myelination. Dev. Cell 32 , 447–458 (2015).

Nave, K. A. Myelination and the trophic support of long axons. Nat. Rev. Neurosci. 11 , 275–283 (2010).

Elitt, M. S. et al. Suppression of proteolipid protein rescues Pelizaeus-Merzbacher disease. Nature 585 , 397–403 (2020).

Chang, A., Tourtellotte, W. W., Rudick, R. & Trapp, B. D. Premyelinating oligodendrocytes in chronic lesions of multiple sclerosis. N. Engl. J. Med. 346 , 165–173 (2002).

Jakel, S. et al. Altered human oligodendrocyte heterogeneity in multiple sclerosis. Nature 566 , 543–547 (2019).

Silbereis, J. C., Pochareddy, S., Zhu, Y., Li, M. & Sestan, N. The cellular and molecular landscapes of the developing human central nervous system. Neuron 89 , 248–268 (2016).

Giedd, J. N. et al. Brain development during childhood and adolescence: a longitudinal MRI study. Nat. Neurosci. 2 , 861–863 (1999).

Breinlinger, S. et al. Hunting the eagle killer: a cyanobacterial neurotoxin causes vacuolar myelinopathy. Science 371 , 6536 (2021).

Article   Google Scholar  

Klose, J. et al. Neurodevelopmental toxicity assessment of flame retardants using a human DNT in vitro testing battery. Cell Biol. Toxicol. 38 , 781–807 (2022).

Lager, A. M. et al. Rapid functional genetics of the oligodendrocyte lineage using pluripotent stem cells. Nat. Commun. 9 , 3708 (2018).

Najm, F. J. et al. Drug-based modulation of endogenous stem cells promotes functional remyelination in vivo. Nature 522 , 216–220 (2015).

Najm, F. J. et al. Rapid and robust generation of functional oligodendrocyte progenitor cells from epiblast stem cells. Nat. Methods 8 , 957–962 (2011).

Richard, A. M. et al. ToxCast chemical landscape: paving the road to 21st century toxicology. Chem. Res. Toxicol. 29 , 1225–1251 (2016).

Sommers, K. J. et al. Quaternary phosphonium compounds: an examination of non-nitrogenous cationic amphiphiles that evade disinfectant resistance. ACS Infect. Dis. 8 , 387–397 (2022).

Hora, P. I., Pati, S. G., McNamara, P. J. & Arnold, W. A. Increased use of quaternary ammonium compounds during the SARS-CoV-2 pandemic and beyond: consideration of environmental implications. Environ. Sci. Tech. Let. 7 , 622–631 (2020).

Article   CAS   Google Scholar  

Takeda, R. et al. Antiviral effect of cetylpyridinium chloride in mouthwash on SARS-CoV-2. Sci. Rep. 12 , 14050 (2022).

Xiao, S., Yuan, Z. & Huang, Y. Disinfectants against SARS-CoV-2: a review. Viruses 14 , 1721 (2022).

Costa-Mattioli, M. & Walter, P. The integrated stress response: from mechanism to disease. Science 368 , eaat5314 (2020).

Madhavan, M. et al. Induction of myelinating oligodendrocytes in human cortical spheroids. Nat. Methods 15 , 700–706 (2018).

Pașca, S. P. et al. A nomenclature consensus for nervous system organoids and assembloids. Nature 609 , 907–910 (2022).

Paul-Friedman, K. et al. Limited chemical structural diversity found to modulate thyroid hormone receptor in the Tox21 chemical library. Environ. Health Perspect. 127 , 97009 (2019).

Liu, W. et al. Prenatal exposure to halogenated, aryl, and alkyl organophosphate esters and child neurodevelopment at two years of age. J. Hazard. Mater. 408 , 124856 (2021).

Hoffman, K., Gearhart-Serna, L., Lorber, M., Webster, T. F. & Stapleton, H. M. Estimated tris(1,3-dichloro-2-propyl) phosphate exposure levels for US infants suggest potential health risks. Environ. Sci. Technol. Lett. 4 , 334–338 (2017).

Ciesielski, T. et al. Cadmium exposure and neurodevelopmental outcomes in U.S. children. Environ. Health Perspect. 120 , 758–763 (2012).

Kwon, S. & O’Neill, M. Socioeconomic and familial factors associated with gross motor skills among US children aged 3–5 years: the 2012 NHANES National Youth Fitness Survey. Int J. Environ. Res Public Health 17 , 4491 (2020).

Steadman, P. E. et al. Disruption of oligodendrogenesis impairs memory consolidation in adult mice. Neuron 105 , 150–164.e6 (2020).

McKenzie, I. A. et al. Motor skill learning requires active central myelination. Science 346 , 318–322 (2014).

Xiao, L. et al. Rapid production of new oligodendrocytes is required in the earliest stages of motor-skill learning. Nat. Neurosci. 19 , 1210–1217 (2016).

Kleinstreuer, N. C. et al. Phenotypic screening of the ToxCast chemical library to classify toxic and therapeutic mechanisms. Nat. Biotechnol. 32 , 583–591 (2014).

Hogberg, H. T. et al. Organophosphorus flame retardants are developmental neurotoxicants in a rat primary brainsphere in vitro model. Arch. Toxicol. 95 , 207–228 (2021).

Renner, H. et al. Cell-type-specific high throughput toxicity testing in human midbrain organoids. Front Mol. Neurosci. 14 , 715054 (2021).

Chesnut, M., Hartung, T., Hogberg, H. & Pamies, D. Human oligodendrocytes and myelin in vitro to evaluate developmental neurotoxicity. Int. J. Mol. Sci. 22 , 7929 (2021).

About List N: Disinfectants for Coronavirus (COVID-19) (United States Environmental Protection Agency, 2023); https://www.epa.gov/coronavirus/about-list-n-disinfectants-coronavirus-covid-19-0

Zheng, G., Webster, T. F. & Salamova, A. Quaternary ammonium compounds: bioaccumulation potentials in humans and levels in blood before and during the Covid-19 pandemic. Environ. Sci. Technol. 55 , 14689–14698 (2021).

Lin, W. & Popko, B. Endoplasmic reticulum stress in disorders of myelinating cells. Nat. Neurosci. 12 , 379–385 (2009).

Li, D., Sangion, A. & Li, L. Evaluating consumer exposure to disinfecting chemicals against coronavirus disease 2019 (COVID-19) and associated health risks. Environ. Int. 145 , 106108 (2020).

Herron, J. M. et al. Multiomics investigation reveals benzalkonium chloride disinfectants alter sterol and lipid homeostasis in the mouse neonatal brain. Toxicol. Sci. 171 , 32–45 (2019).

Patisaul, H. B. et al. Beyond cholinesterase inhibition: developmental neurotoxicity of organophosphate ester flame retardants and plasticizers. Environ. Health Perspect. 129 , 105001 (2021).

Hou, M., Zhang, B., Fu, S., Cai, Y. & Shi, Y. Penetration of organophosphate triesters and diesters across the blood–cerebrospinal fluid barrier: efficiencies, impact factors, and mechanisms. Environ. Sci. Technol. 56 , 8221–8230 (2022).

Blum, A. et al. Organophosphate ester flame retardants: are they a regrettable substitution for polybrominated diphenyl ethers? Environ. Sci. Technol. Lett. 6 , 638–649 (2019).

Gibson, E. A. et al. Flame retardant exposure assessment: findings from a behavioral intervention study. J. Expo. Sci. Environ. Epidemiol. 29 , 33–48 (2019).

Najm, F. J. et al. Isolation of epiblast stem cells from preimplantation mouse embryos. Cell Stem Cell 8 , 318–325 (2011).

Najm, F. J. et al. Transcription factor-mediated reprogramming of fibroblasts to expandable, myelinogenic oligodendrocyte progenitor cells. Nat. Biotechnol. 31 , 426–433 (2013).

Allan, K. C. et al. Non-canonical targets of HIF1a impair oligodendrocyte progenitor cell function. Cell Stem Cell 28 , 257–272.e11 (2021).

Hubler, Z. et al. Accumulation of 8,9-unsaturated sterols drives oligodendrocyte formation and remyelination. Nature 560 , 372–376 (2018).

Bankhead, P. et al. QuPath: open source software for digital pathology image analysis. Sci. Rep. 7 , 16878 (2017).

Patro, R., Duggal, G., Love, M. I., Irizarry, R. A. & Kingsford, C. Salmon provides fast and bias-aware quantification of transcript expression. Nat. Methods 14 , 417–419 (2017).

Love, M. I., Huber, W. & Anders, S. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome Biol. 15 , 550 (2014).

Adamson, B. et al. A multiplexed single-cell CRISPR screening platform enables systematic dissection of the unfolded protein response. Cell 167 , 1867–1882.e21 (2016).

Wong, Y. L. et al. eIF2B activator prevents neurological defects caused by a chronic integrated stress response. eLife 8 , e42940 (2019).

Judson, R. et al. Analysis of the effects of cell stress and cytotoxicity on in vitro assay activity across a diverse chemical and assay space. Toxicol. Sci. 153 , 409 (2016).

Paul Friedman, K. et al. Utility of in vitro bioactivity as a lower bound estimate of in vivo adverse effect levels and in risk-based prioritization. Toxicol. Sci. 173 , 202–225 (2020).

Download references

Acknowledgements

This work was supported by National Institutes of Health grants R35NS116842 (P.J.T.), F31NS124282 (E.F.C.), T32NS077888 (E.F.C.) and T32GM007250 (E.F.C.). B.L.L.C. is supported by an NMSS Career Transition Fellowship. M.A.S. received support from the Howard Hughes Medical Institute Hanna H. Gray Fellowship and the New York Stem Cell Foundation Druckenmiller Fellowship. Institutional support was provided by CWRU School of Medicine, and philanthropic support was generously contributed by sTF5 Care and the Long, Walter, Peterson, Goodman and Geller families. The funders had no role in the study design, data collection and analysis, decision to publish or preparation of the manuscript. Additional support was provided by the Small Molecule Drug Development and Light Microscopy Imaging core facilities of the CWRU Comprehensive Cancer Center (P30CA043703). The EPA provided the ToxCast screening library through an MTA with CWRU and supported the effort of EPA employees (T.J.S. and K.P.F.). We are grateful to D. Adams, A. Wynshaw-Boris, D. Kassel, K. Carr, J. Kristell, K. Allan, E. Shick, R. Ziar, A. Sterling and A. Gartley for technical assistance and/or discussion and C. Lilliehook for editorial support. This work was supported in part by the EPA and has been reviewed and approved for publication by the EPA’s Center for Computational Toxicology and Exposure. Approval for publication does not signify that the contents reflect the views of the Agency, nor does mention of trade names or commercial products constitute an endorsement or recommendation for use.

Author information

Authors and affiliations.

Department of Genetics and Genome Sciences, Case Western Reserve University School of Medicine, Cleveland, OH, USA

Erin F. Cohn, Benjamin L. L. Clayton, Mayur Madhavan, Kristin A. Lee, Sara Yacoub, Yuriy Fedorov, Marissa A. Scavuzzo & Paul J. Tesar

Center for Computational Toxicology and Exposure, Office of Research and Development, U.S. Environmental Protection Agency, Research Triangle Park, NC, USA

Katie Paul Friedman & Timothy J. Shafer

You can also search for this author in PubMed   Google Scholar

Contributions

E.F.C., B.L.L.C., T.J.S. and P.J.T. conceived this study to screen for the effects of environmental chemicals on oligodendrocyte development. E.F.C., B.L.L.C. and P.J.T. designed and managed the experimental studies. E.F.C., B.L.L.C., K.A.L. and S.Y. performed, quantified and analyzed in vitro experiments using mouse OPCs, including primary screening and immunocytochemistry. E.F.C., K.A.L. and S.Y. performed dose–curve validations and qPCR. B.L.L.C. isolated mouse astrocytes and performed primary screening for astrocytes. E.F.C. performed RNA-seq analysis. E.F.C., K.A.L. and M.A.S. performed all in vivo experiments. K.P.F. performed ToxPrint chemotype enrichment analyses, and T.J.S. and K.P.F. guided categorization of chemical screen hits. E.F.C. designed and performed linear regression analyses using data from the NHANES. M.M. and E.F.C. performed cortical organoid experiments. Y.F. managed the chemical library and pipelined primary screening data. E.F.C. assembled all figures. E.F.C. and P.J.T. wrote the manuscript with input from all authors.

Corresponding author

Correspondence to Paul J. Tesar .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Neuroscience thanks Lucas Cheadle, Thomas Hartung, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended data fig. 1 screening a library of environmental chemicals in developing oligodendrocytes identifies cytotoxic chemicals and modulators of oligodendrocyte generation..

Representative heatmaps of one of 6 primary screening 384-well plates depicting cytotoxic compounds (red), oligodendrocyte inhibitors (blue), and drivers (yellow). Viability and percent O1+ oligodendrocytes are normalized to DMSO vehicle control (-). Thyroid hormone, a known driver of oligodendrocyte generation, is included as a positive control (+) for oligodendrocyte development. Cytotoxic hits (gray) were omitted from heatmap displaying normalized O1 percentage. b . Quantification of hits across 6 primary screening plates showing distribution of chemicals identified as cytotoxic (black), drivers (green), and inhibitors (blue). c . Top use categories for 206 validated cytotoxic chemicals and the number of chemicals belonging to each category. d . Venn diagram showing the overlap of 206 validated cytotoxic chemicals identified in the oligodendrocyte screen compared to cytotoxic hits identified in an identical screen performed in mouse astrocytes.

Source data

Extended data fig. 2 quaternary compounds are selectively cytotoxic to oligodendrocytes through activation of the integrated stress response..

a-b . mPSC-derived oligodendrocytes and primary mouse oligodendrocytes were treated with 20 µM methyltrioctylammonium chloride and tributyltetradecylphosphonium chloride. a . Representative images showing DAPI and O1 immunostaining. Scale bar, 50 μm. b . Quantification of viability normalized to DMSO. Data are mean ± SD, n = 3 biological replicates. c-h . Viability of mouse PSC-derived oligodendrocytes, primary astrocytes, and fibroblasts, normalized to DMSO after treatment with quaternary compounds. Data are mean ± SEM, n = 3 biological replicates. i . IC 50 concentrations of quaternary compounds in mouse oligodendrocytes, astrocytes, and fibroblasts, n = 3 biological replicates. j . Viability of oligodendrocytes normalized to DMSO cultured in the presence of methyltrioctylammonium chloride, ADEBC (C12-C14), or cetylpyridinium chloride at their respective IC 75 and QVD-OPH, necrostatin-1, and ferrostatin-1. Data are mean, n = 3 biological replicates k,l . Volcano plot of differentially expressed genes in oligodendrocytes treated with 370 nM (IC 75 ) ADEBC (C12-C14) ( k ) or 181 nM (IC 75 ) cetylpyridinium chloride ( l ) for 4 hours. Log2FC and padj were calculated with DESeq2. Genes highlighted in red increased (padj ≤ 0.05), n = 3 biological replicates. Top 10 genes with the greatest Log2FC are labelled. m . qRT-PCR of CHOP in oligodendrocytes treated with DMSO, or top toxic compounds identified in the primary screen (not quaternary compounds). Oligodendrocytes were cultured for 4 hours in the presence of chemicals at IC 75 or 20 µM if the calculated IC 75 exceeded the primary screening concentration (388 nM Basic Blue 7, 20 µM 3,3’-dimethylbenzidine, 7.14 µM diisononyl cyclohexane-1,2-dicarboxylate, 1.82 µM 3,3’-dimethoxybenzidine, or 20 µM 2,4-dimethylphenol). Data are mean ± SD, n = 3 biological replicates. p-values calculated using one-way ANOVA with Dunnett post-test correction for multiple comparisons. n . qRT-PCR of CHOP in fibroblasts treated for 4 hours normalized to DMSO. Quaternary compounds were tested at their IC 75 (calculated from dose response testing in fibroblasts) or 20 µM if IC 75  > 20 µM (2.0 µM methyltrioctylammonium chloride, 18.8 µM ADEBC (C12-C14), 20 µM cetylpyridinium chloride). Data are mean ± SD, n = 3 biological replicates. p-values calculated using one-way ANOVA with Dunnett post-test correction for multiple comparisons. o . Oligodendrocyte viability (normalized to DMSO) after treatment with quaternary compounds (IC 75 ) and JSH-23 or Pifithrin-µ. Data shown as mean, n = 3 biological replicates.

Extended Data Fig. 3 Quaternary compounds are toxic to mouse oligodendrocytes in vivo and in human cortical organoids.

a,b . Brain and liver concentration of methyltrioctylammonium chloride, ADEBC (C12-C14), and cetylpyridinium chloride after oral gavage (P9-P10). Data are mean ± SD, n = 3 or 1 mice (100 mg/kg/day methyltrioctylammonium chloride mice presented from n = 1 mouse due to lethality to other mice included in the study). c . Survival of mice treated with vehicle or cetylpyridinium chloride, n = 8 (vehicle), n = 10 (10 mg/kg/day), n = 11 (1 mg/kg/day). Mice were considered dead if found dead in their cage or cannibalized by dam. d-k . Mice were gavaged P5-P14 with vehicle (water) or 1 mg/kg/day cetylpyridinium chloride. Measurement of daily body ( d ), P14 liver ( e ), and P14 brain ( f ) weights. g . P14 cetylpyridinium chloride liver concentration, with analyte concentrations below the lower limit of detection (1 ng/mL) coded as 0. h . Representative images showing DAPI and SOX10 immunostaining. i . Quantification of oligodendrocyte lineage cell density (SOX10+ per mm 2 ) in cortex and hippocampus of P14 mice. j . Representative images showing DAPI and NEUN immunostaining. k . Quantification of neuron density (NEUN+ per mm 2 ) in cerebellum, cortex, and hippocampus of P14 mice. Data are mean ± SD, n = 8 or 9 mice. p-values calculated using unpaired two-tailed t test ( e, f, i, k ). Scale bars, 50 μm ( h, j ). l-o . Human cortical organoids were treated with DMSO, 94 nM methyltrioctylammonium chloride, 370 nM ADEBC (C12-C14), or 181 nM cetylpyridinium chloride (IC 75 ). l . Representative images showing immunostaining of oligodendrocyte lineage cells (SOX10 + ), progenitors (SOX2 + ), and neurons (NeuN + ). Scale bar, 50 μm. Quantification of pre-OPC (SOX2 + SOX10+ per mm 2 ) ( m ), other progenitor (SOX2 + SOX10- per mm 2 ) ( n ), and neuron (NeuN+ per mm 2 ) ( o ) densities in whole cortical organoids. Data are mean ± SD, n = 22, 24, 29, or 30 biological replicates (individual organoids from 4 independent batches), colored based on batch. p-values calculated using one-way ANOVA with Dunnett post-test correction for multiple comparisons.

Extended Data Fig. 4 Organophosphate flame retardants inhibit the development of mouse oligodendrocytes in vitro and in vivo .

a . Primary chemical screen of 1,531 non-cytotoxic environmental chemicals showing the effect of individual chemicals on oligodendrocyte generation, presented as percent O1+ cells normalized to the DMSO control, as shown in Fig. 2a . Dotted lines show the hit cutoffs for drivers and inhibitors. Drivers increase O1+ percentage by 22% ( > 3 SDs). Inhibitors reduce O1+ percentage by more than 50% ( > 7 SDs). Thyroid modulators are highlighted in yellow. b . IC 50 concentrations, cytotoxicity median values, and use categories for three organophosphate esters identified as inhibitors of oligodendrocyte development, n = 3 biological replicates. c-d . mPSC-derived OPCs and mouse primary OPCs were treated with organophosphate flame retardants (20 μM). c . Representative images showing DAPI and O1 immunostaining, scale bar, 50 μm. d . Quantification of oligodendrocytes (O1 + ). e-f . mPSC-derived OPCs were treated with 20 µM TBPP or TMPP. Representative images ( e ) and quantification ( f ) of early (O4 + ), intermediate (O1 + ), and late (MBP + ) oligodendrocytes. Control images and TDCIPP treated oligodendrocytes are shown in Fig. 2e . Nuclei are marked with DAPI. Scale bar, 50 μm ( e ). Data are mean ± SEM, n = 3 biological replicates p-values calculated using two-way ANOVA (ANOVA p = ) for overall chemical differences with Dunnett’s multiple comparison test for differences within each time point (p =) ( f ). g-l . Mice were treated with vehicle (corn oil), 10 mg/kg/day, or 100 mg/kg/day TDCIPP. Measurement of P14 body ( g ), brain ( h ), and liver ( i ) weights. j . TDCIPP liver concentrations at P14, with analyte concentrations below lower limit of detection (10 ng/mL) coded as 0. k . Representative images showing DAPI, SOX10, and CC1 immunostaining Scale bar, 50 μm. l . Quantification of oligodendrocyte density (SOX10 + CC1+ per mm 2 ) in the hippocampus and cortex of P14 mice. Data are mean ± SD from n = 8 or 9 mice ( g-j, l) . p-values were calculated using one-way ANOVA with Dunnett post-test correction for multiple comparisons ( g-i, l ).

Extended Data Fig. 5 TDCIPP is associated with abnormal neurodevelopmental outcomes in children.

a . Representative images of human cortical organoids treated with 18.7 µM TDCIPP for 10 days, showing DAPI and NEUN immunostaining. Scale bar, 50 μm. b . Quantification of neurons (NEUM+ per mm 2 ) in whole cortical organoids. Data are mean ± SD from n = 21 or 29 biological replicates (individual organoids from 4 independent batches). Data points are colored based on organoid batch. p-values calculated using unpaired two-tailed t test. c . Venn diagram showing co-occurrence of two neurodevelopmental outcomes in the study population. d-e . Adjusted odds ratio and associated p-values for covariates used in the logistic regression model for the neurodevelopmental outcomes requiring special education and gross motor dysfunction, n = 1564 or 1566. Significant odds ratios (p-value ≤ 0.05) are indicated by closed circles. Closed or open circles are the odds ratio and error bars indicate the 95% CI. Odds ratios and p-values were generated and visualized with the “survey” and “gtsummary” R packages. The Wald test was used to calculate p-values.

Supplementary information

Reporting summary, supplementary table 1 - primary screening results., supplementary table 2 -toxprint enrichment analysis., supplementary table 3 - gene set enrichment analysis., source data fig. 1.

Source data for graphs in Fig. 1.

Source Data Fig. 2

Source data for graphs in Fig. 2.

Source Data Fig. 3

Source data for graphs in Fig. 3.

Source Data Fig. 4

Source data for graphs in Fig. 4.

Source Data Extended Data

Source data for graphs in Extended Data Fig. 1.

Source data for graphs in Extended Data Fig. 2.

Source data for graphs in Extended Data Fig. 3.

Source data for graphs in Extended Data Fig. 4.

Source data for graphs in Extended Data Figure 5.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Cohn, E.F., Clayton, B.L.L., Madhavan, M. et al. Pervasive environmental chemicals impair oligodendrocyte development. Nat Neurosci (2024). https://doi.org/10.1038/s41593-024-01599-2

Download citation

Received : 15 November 2022

Accepted : 05 February 2024

Published : 25 March 2024

DOI : https://doi.org/10.1038/s41593-024-01599-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Open access
• Published: 29 March 2024

Development of a nomogram based on the clinicopathological and CT features to predict the survival of primary pulmonary lymphoepithelial carcinoma patients

• Kai Nie 1 ,
• Lin Zhu 1 ,
• Yuxuan Zhang 2 ,
• Yinan Chen 1 ,
• John Parrington 2 &
• Hong Yu 1  

Respiratory Research volume  25 , Article number:  144 ( 2024 ) Cite this article

27 Accesses

Metrics details

The aim of this study was to develop a nomogram by combining chest computed tomography (CT) images and clinicopathological predictors to assess the survival outcomes of patients with primary pulmonary lymphoepithelial carcinoma (PLEC).

113 patients with stage I–IV primary PLEC who underwent treatment were retrospectively reviewed. The Cox regression analysis was performed to determine the independent prognostic factors associated with patient’s disease-free survival (DFS) and cancer-specific survival (CSS). Based on results from multivariate Cox regression analysis, the nomograms were constructed with pre-treatment CT features and clinicopathological information, which were then assessed with respect to calibration, discrimination and clinical usefulness.

Multivariate Cox regression analysis revealed the independent prognostic factors for DFS were surgery resection and hilar and/or mediastinal lymphadenopathy, and that for CSS were age, smoking status, surgery resection, tumor site in lobe and necrosis. The concordance index (C‑index) of nomogram for DFS and CSS were 0.777 (95% CI: 0.703–0.851) and 0.904 (95% CI: 0.847–0.961), respectively. The results of the time‑dependent C‑index were internally validated using a bootstrap resampling method for DFS and CSS also showed that the nomograms had a better discriminative ability.

Conclusions

We developed nomograms based on clinicopathological and CT factors showing a good performance in predicting individual DFS and CSS probability among primary PLEC patients. This prognostic tool may be valuable for clinicians to more accurately drive treatment decisions and individualized survival assessment.

Primary pulmonary lymphoepithelial carcinoma (PLEC) is a unique and rare subtype of non-small-cell lung cancer (NSCLC), accounting for less than 0.7% of all NSCLCs [ 1 , 2 , 3 ]. PLEC was first reported in 1987 and histologically resembles undifferentiated nasopharyngeal carcinoma (NPC) [ 4 ]. From the epidemiological and etiological perspective, PLEC is more common in Asian ethnicities, tends to occur in relatively young and middle-aged individuals, and is generally considered to be closely related to Epstein–Barr virus (EBV) infection [ 5 ]. PLEC was previously classified as a subtype of large-cell lung cancer [ 6 ]. In 2015, the World Health Organization (WHO) classified it as one of the “other and unclassified carcinomas” [ 7 ]. In contrast, the latest 5th edition of the WHO classification of thoracic tumors in 2021 re-categorized PLEC as a subtype of squamous cell carcinomas (SCCs) [ 8 ]. The constantly changing classification of PLEC indicates the imperative need for further research.

Currently, the treatment of PLEC mainly follows the NCCN Clinical Practice Guidelines for NSCLC [ 9 , 10 ]. Due to the rarity of primary PLEC, the standard of management for this disease is not still established [ 9 ]. In particular, mutations of commonly driven genes are lacking for PLEC patients, and targeted therapy drugs have little significance [ 11 ]. Therefore, the lack of treatment methods and experience for treating PLEC patients indicates an imperative need for individualized clinical management and precise survival prediction.

In a prognostic setting, the estimation of risk probability is rarely based on individual risk factors, as reliable estimates are insufficient. Discovering more prognostic factors and estimating based on multivariate models are now considered more reliable methods. Chest computed tomography (CT) is the routine imaging method for lung cancer detection and post-treatment management, and the CT image features have significant value in the diagnosis and prognosis of lung cancer [ 12 , 13 ]. However, the relevant studies on imaging characteristics of primary PLEC are very few, and the cohort size of each published study was quite small [ 13 , 14 , 15 , 16 ]. Several reports have integrated clinical and pathological data from several PLEC patients for prognostic evaluation [ 17 , 18 ], but the CT imaging features associated with the survival outcome of primary PLEC have not yet been described.

Therefore, this study aimed to develop a model including clinicopathological and CT features to estimate the disease-free survival (DFS) and cancer-specific survival (CSS) in patients with primary PLEC and to evaluate its clinical predictive ability and net benefit rate for individual survival estimation.

This retrospective study was approved by the institutional review board (No. IS22019), and the requirement for written informed consent was waived.

This study was conducted in patients with pathologically diagnosed PLEC between October 2009 and March 2023 at Shanghai Chest Hospital Affiliated to the Shanghai Jiao Tong University School of Medicine (Shanghai, China). In total, 141 cases were initially retrospectively recruited. The inclusion criteria were as follows: (1) CT scan was performed before treatment; (2) The diagnosis of primary PLEC was confirmed by fine-needle biopsy or complete surgical resection pathology; (3) The patient’s baseline characteristics and clinical data were complete. The exclusion criteria were as follows: (1) The past history of other malignancy, and (2) Metastasis of nasopharyngeal PLEC. Finally, A total of 113 patients were included in this study (51 males and 62 females; mean age, 56.8 years ± 11.5; range, 20–81 years). Figure  1 shows the patient recruitment pathway, along with the exclusion criteria. All primary PLEC tumors were reclassified based on the 5th edition of the WHO classification of Thoracic Tumors. Tumor staging was performed based on the American Joint Committee on Cancer TNM Staging Manual, 8th edition [ 19 ]. Among the 113 patients, 85 patients who underwent surgery provided pathological stage, while the remaining 28 patients who underwent non-surgical treatment provided clinical stage alone. We reviewed clinicopathological records and pre-treatment CT imaging data of all patients.

Flowchart shows patient selection

Imaging examination protocol

Among the 113 patients, 34 patients underwent a plain chest CT, 79 cases underwent both unenhanced and enhanced CT. Somatom Definition AS (Siemens Medical Systems, Erlangen, Germany) and Brilliance 40 (Philips Medical Systems, the Netherlands Cleveland, state of Ohio, USA) scanners were used as the scanning machine. Patients were scanned at the end of inspiration during a single breath hold in the supine position. CT settings were as follows: tube voltage, 120 kVp; average tube current, 250 mA; pitch, 0.984; and section thickness, 1 mm. Scans covered the region from the top of the thoracic cage to the level of bilateral adrenal glands, and patients underwent a contrast-enhanced CT scan (non-ionic contrast medium, 60–80 mL). All imaging data were reconstructed using the standard algorithm and viewed with both lung window (window width, 1,500 HU; window level, − 500 HU) and mediastinal window (window width, 350 HU; window level, 50 HU).

Image analysis

All post-processed images were interpreted retrospectively and independently by two experienced thoracic radiologists (HY and YNC) with 10 and 30 years of experience in chest imaging. The observers were blinded to the identities and clinical data of the patients. For all disagreements between the two observers on CT findings, the decisions were then reached by consensus. The location, shape, size, margin, interface, internal features, adjacent structures and CT attenuation values of the lesion were assessed. The definitions and scoring rules of morphological features are described in Table S1 .

CT, MRI, or PET/CT imaging was performed for the post-treatment disease status evaluation, and patients were evaluated once every six months within the first two years and then annually thereafter unless a specific clinical event emerged. The primary endpoint was DFS, which was defined from the date of initial histological diagnosis to the date of the first recorded evidence of clinical recurrence or distant metastasis as confirmed by histological evidence or death by any related causes. The secondary endpoint was CSS, calculated from the initial histological diagnosis to the date of death resulting from the progression of lung cancer (local and/or distant). The patient’s medical records and a telephone consultation were used for follow-up.

Statistical analysis

Statistical analysis was performed using SPSS software, version 26.0 (SPSS Inc., Chicago, IL, USA), R software, version 3.0.1 ( http://www.R-project.org ) and X-tile software, version 3.6.1 (Yale University School of Medicine, New Haven, Conn). The nomogram, decision curve analysis curves and calibration curves were plotted by the rms package in R. Survival curve was plotted using Kaplan–Meier survival analysis and compared using the log-rank test with the survminer and survival package in R. Continuous variables are summarized as means and standard deviations if the distribution was normal or as medians and interquartile range (IQR) if the distribution was not normal. Categorical variables are reported as frequencies and percentages. Two-tailed p  < 0.05 was considered statistically significant.

In this study, CT values were transformed into categorical variables and the optimal cut-off values were obtained by X-tile [ 20 ]. The repeatability for quantitative tumor size measurement was analyzed using the intraclass correlation coefficient (ICC). Reproducibility was defined as poor (ICC intraclass correlation coefficient < 0.75), moderate (ICC intraclass correlation coefficient = 0.75–0.90), or high (ICC intraclass correlation coefficient > 0.90) [ 21 ]. Interobserver agreement for qualitative variables of CT imaging was evaluated using Cohen’s kappa analysis. The κ value was interpreted as < 0.20, poor or slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, good agreement; and 0.81–1.00, very good agreement [ 22 ].

Predictors for DFS and CSS were selected by Cox proportional hazards regression analysis. As PLEC is a rare tumor and the number of cases is not many and the events is less. In addition, CT findings of PLEC is a new insight and the prognostic analysis is exploring, therefore, those with a significant level of p  ≤ 0.05 in univariate analysis and statistically insignificant but clinically significant were entered into the multivariate Cox regression method with a backward stepwise selection procedure. A nomogram with endpoints of 3- and 5-year CSS and DFS were constructed based on the multivariate Cox regression analysis results, respectively. Harrell’s concordance index (C-index) was measured to quantify the discriminative performance of nomograms. All internal validations were performed using a bootstrapping method with 500 resamples. The calibration curves of nomogram were then drawn for the 3-year and 5-year CSS and DFS of the patients, which illustrated both survival probabilities predicted by nomogram and the observed probabilities. The decision curve analysis was conducted to estimate the clinical usefulness of the nomogram by quantifying the net benefits at different threshold probabilities. Finally, subjects were divided into high- and low-risk groups according to the median on the nomogram scores obtained from the constructed model. The Kaplan–Meier method and log-rank test were applied to calculate and compare risk group differences. Data between groups were compared using the independent t-test. Furthermore, categorical variables were presented with count (%) and were compared using the χ 2 test.

Patients baseline characteristics

The clinicopathological features of all PLEC patients are shown in Table  1 . In the primary PLEC cohort, the median follow-up time was 53.1 months (range: 1–157.4 months). The DFS and CSS of all PLEC patients are shown in Fig.  2 and Fig S1 a. The median DFS and CSS was not reached. The 1-, 3-, and 5-year CSS rates were 99.0, 88.6 and 76.1%, respectively. The 1-, 3-, and 5-year DFS rates were 88.4, 68.2, and 60.4%, respectively. The optimal cut-off value for CT attenuation was 37.8 HU which was obtained by X-tile. The ICC for the quantitative measurement of tumor size was 0.997 (95% confidence interval [CI]: 0.995–0.998; P  < 0.001). The interobserver reproducibility for qualitative CT imaging features was good or excellent (κ, 0.73–1.00). Table S2 showed a detailed description of the inter–reader agreement. The detailed CT features of the 113 patients are summarized in Table  2 .

KaplanMeier curve for DFS of total patients

Developing a clinicopathological and CT imaging-based nomogram to predict DFS and CSS

The results of the univariate and multivariate Cox analysis for predictive factors are presented in Table  3 and Table S3 . According to multivariate analysis results for DFS, a total of four variables were retained through backward stepwise selection; only the surgery resection ( p  = 0.001, HR = 0.24; 95% CI 0.11–0.55) and Hilar and/or mediastinal LAP ( p  = 0.007, HR = 3.27; 95% CI 1.39–7.70) being significant independent prognostic factors. According to multivariate analysis results for CSS, a total of six variables were retained, and the following variables showed significantly independent prognostic factors: age ( p  < 0.001, HR = 1.13; 95% CI 1.06–1.20), smoking status ( p  = 0.038, HR = 4.15; 95% CI 1.09–15.88), surgery resection ( p  < 0.001, HR = 0.05; 95% CI 0.01–0.19), tumor site in lobe ( p  = 0.014, HR = 0.29; 95% CI 0.11–0.78), hilar and/or mediastinal LAP ( p  = 0.038, HR = 4.49; 95% CI 1.09–18.53) and necrosis ( p  = 0.011, HR = 3.96; 95% CI 1.37–11.50). The HRs and 95% CIs for the multivariate Cox regression analysis for remaining DFS and CSS risk factors are shown as forest plots in Fig.  3 a and Fig. S1 b. Consequently, the nomograms for predicting the probability of 3-and 5-year DFS and CSS of all primary PLECs were developed using the risk factors combined with clinical and CT Imaging features (Fig.  3 b and Fig. S1 c). To use the nomogram, a vertical line needs to be delineated to the point raw to assign point values for each factor, and the total points are calculated as the sum of the risk points of all risk factors.

The forest plot of factors obtained through multivariate COX regression analysis for DFS (a) ; The nomogram established for prediction of DFS (b)

The discrimination, net benefit and predictive capacity of the nomogram

The C-indexes of the nomograms for DFS and CSS prediction in the dataset were 0.777 (95% CI: 0.703–0.851) and 0.904 (95% CI: 0.847–0.961), respectively. The performance of nomogram for clinical prediction was evaluated using the area under the receiver-operating characteristic (ROC) curve (AUC) (Fig.  4 a and Fig. S1 d), the 3- and 5-year AUC for DFS were 0.820 and 0.901, respectively, and those for CSS were 0.941 and 0.922, respectively. Moreover, time-dependent C-index analysis also showed that the nomograms exhibited good prognostic accuracy in clinical outcome prediction for DFS or CSS. A similar result was also observed in internal validation using a bootstrap resampling method (red lines) (Fig.  4 b and Fig. S1 e). The calibration plots of the prognostic nomograms in predicting 3- and 5-year DFS and CSS demonstrated good coincidences between the estimated risk and observed risk (Fig.  5 a and Fig. S1 f). The decision curve analysis for 3-and 5-year DFS and CSS showed that the combined nomogram had a higher overall net benefit than each clinical and CT imaging factor across the majority of the range of reasonable threshold probabilities (Fig.  5 b,c and Fig. S1 g,h).

Area under the curves at 3-year and 5-year were calculated to assess the prognostic accuracy for DFS (a) ; Timedependent Cindex of nomogram of all PLEC patient (blue lines) and internally validated using a bootstrap resampling method (red lines) for DFS (b)

Calibration curves for 3, 5year DFS (a) of nomogram predictions; Decision curve analysis of nomogram for 3year DFS (b) and 5year DFS (c) of PLEC patients. The red line is the net benefit of a strategy of treating all people; the brown line is the net benefit of treating no people. The yaxis indicates the overall net benefit, which is calculated by summing the benefits (true-positive results) and subtracting the harms (false-positive results), weighting the latter by a factor related to the relative harm of undetected cancer compared with the harm of unnecessary treatment

Risk stratification for PLEC patients

To assess whether the primary PLEC patients could be effectively separated into two proposed risk groups based on the nomograms, we calculated each patient’s total point and used the median to determine the optimal cut-off value. Patients with nomogram scores less than or equal to the median were classified as low-risk groups, and those with scores greater than the median were classified as high-risk groups. According to the range of total points, the Kaplan- Meier curves highlighted the appropriateness of distinguishing the patients’ survival for DFS and CSS in all the subgroups. The groups were obtained considering the total point distribution of our cohort. Compared with the high-risk group (red lines), group low-risk (blue lines) represent patients with better prognoses (Fig.  6 and Fig. S1 i). In order to explore individual factor comparisons within the clinical, pathologic, and chest CT factors between the high-risk and low-risk groups, we conducted a statistical comparison of various risk factors for patients with different risks, and the relevant results are shown in Table S4 , 5 .

KaplanMeier curve for DFS based on the nomogram prediction

Discussions

In our cohort, it was found that female patients and non-smokers accounted for the majority. Most patients were found during physical examinations, while a few had symptoms such as cough with or without blood-tinged sputum, similar to other NSCLCs without specificity [ 23 , 24 ]. No common mutation-driving genes in lung cancer were observed in our study. The above characteristics are consistent with the results reported in previous studies [ 2 , 3 , 18 ]. In addition, we restaged 113 PLEC patients in this cohort according to the 8th edition of the TNM staging system. The results showed that nearly half of the PLEC patients have a higher TNM stage (III + IV, 45.2%) at initial diagnosis, indicating that surgical resection is no longer feasible for treatment and requires multidisciplinary collaborative treatment.

Routine initial and follow-up examinations of lung lesions mainly rely on CT scans in clinical practice. Therefore, we evaluated the morphological CT manifestations of 113 patients with primary PLEC before treatment. The results showed that the median maximum diameter on CT imaging was 3.4 cm (IQR, 2–4.7 cm), and the average CT value on plain CT scan was 37.3 ± 10.5 HU. This indicates that PLEC often presents as large, soft tissue-dense masses on CT. Tumors are mostly located in the right lobe of the lung and are more common in peripheral types. However, few studies have reported that PELC mainly manifested as the central type of lung cancer [ 14 , 15 ]; it may be related to the small number of included cases.

Further CT imaging analysis showed that most PLECs exhibit solitary, well-defined solid nodules or masses, with lobulation sign more common, spiculation sign less common, and bronchogram cut-off more common. These characteristics are consistent with previous research results [ 14 , 15 , 25 ]. Moreover, the hilar and/or mediastinal LAP was more common in this cohort (54/113, 47.8%), indicating that primary PLEC is prone to lymph node metastasis. With a summary of these CT scanning characteristics, we attempted to integrate the clinical, pathological, and CT imaging features of all PLEC patients in the cohort. We conducted long-term follow-ups to discover more potential indicators for predicting survival risk.

Based on univariate and multivariate analysis for DFS and CSS, PLEC patients who did not receive surgery had a worse CSS and DFS because patients who have not undergone surgery are often in the advanced stage of TNM staging. On the multivariate analysis, hilar and/or mediastinal LAP was an independent prognostic factor for DFS and CSS. Previous studies also reported that nodal stage in the TNM system and lymph node involvement were independent prognostic factors for post-operative recurrence-free survival (RFS) in stage I-IIIa PLEC patients [ 26 ]. Our findings suggest that as a non-invasive examination method, pre-treatment hilar and/or mediastinal LAP on CT images in stage I-IV PLEC patients can provide an independent value for predicting survival outcomes. On the multivariate analysis for CSS, we found that age, smoking history, tumor site in the lobe, and necrosis signs were independent prognostic factors. Older PLEC patients and those with a history of smoking have a higher risk of death. A previous study found that PLEC patients with lesions in the left lobe of the lung seemed to have a poorer DFS in univariate analysis ( p  = 0.051), but they only included 30 cases of PELC [ 17 ]. Our study expanded the size of the study cohort and covariates, further demonstrating that the location of tumors in PLEC patients was an independent prognostic factor for CSS, indicating that patients with tumors in the left lung lobe have a higher risk of survival. This may be due to the lack or difficulty in 4 L lymph node dissection during routine surgical resection in patients with left lung cancer, resulting in a poorer prognosis compared to right lung cancer patients [ 27 ]. It is worth noting that female lung cancer patients often have better prognosis than males [ 28 ], while in the univariate analysis of this study, the prognosis of females was worse than that of males. We consider this may be due to the small number of included cases. In particular, age and smoking history were not significant in univariate analysis in this cohort ( p  = 0.148, 0.546, respectively) but became independent prognostic factors for CSS when included in multivariate analysis. This fully indicates that age and smoking history, once combined with other prognostic factors, have an impact on the prognosis of PLEC patients.

Furthermore, patients with necrosis on CT images had poorer CSS; this might because necrosis often occurs in large tumors with insufficient blood supply, while larger tumors have higher T staging and poorer prognosis. These conclusions may help clinicians understand the relationship between CT findings and patient survival in PLEC patients. In addition, we also found that symptomatic patients with elevated CYFRA21-1, irregularity shape on CT images, CT values (<37.8 HU as ref.) and patients with pleural and/or pericardial effusion had worse prognosis for both DFS and CSS on univariate analysis ( p <0.05), but not significant on multivariate analysis. This suggests that these variables may potentially correlate with the prognosis of PLEC patients. Especially in this cohort, up to 37.2% of PLEC patients had elevated levels of CYFRA21-1, which had been proven to be highly expressed in SCCs [ 29 ], indirectly demonstrating the necessity for primary PLEC to be classified as a subtype of lung SCCs.

Based on the Cox multivariate regression analysis results, we developed nomograms model that included multiple clinical and CT imaging prognostic factors to predict DFS and CSS in PLEC patients. Our nomograms showed the C-indexes of the overall dataset were higher than 0.7 and AUCs greater than 0.8 under the 3-year and 5-year ROC curves, indicating that the nomograms have an excellent discrimination performance for predicting clinical outcomes. The results of time-dependent C-index analysis further showed that this combined nomogram still had good predictive ability after undergoing 500 resamples of internal bootstrap validation. Moreover, the 3-year and 5-year decision curves and calibration plots for CSS and DFS showed that the nomograms we developed had strong prediction accuracy and overall net benefits and could evaluate clinical relevance without additional validation data in traditional decision analysis methods [ 30 ]. In addition, this nomogram can successfully classify PLEC patients into high and low-risk subgroups. Compared to the low-risk group, the high-risk group had the worst prognosis ( p < 0.001). In summary, our nomogram, which combines pre-treatment CT imaging and clinicopathological features, has great potential in clinical application for predicting the prognosis of PLEC patients and may assist clinicians in the decision-making process, allowing patients to obtain more benefits.

However, our research still has some limitations. Firstly, our research findings are based on a retrospective design; therefore, this study cannot exclude all potential inherent biases. Secondly, our data were obtained from a single cancer center, and the sample size was relatively small, the prediction model of prognosis was sufficient for DFS but for CSS. Finally, we did not find enough samples for external validation.

In conclusion, we first studied the relationship between CT imaging features and the prognosis of primary PLEC patients, and the identified CT imaging features may serve as biomarkers for prognostic risk stratification in PLEC patients. At the same time, we have developed new nomograms that combine clinicopathological and CT imaging features for individualized survival risk assessment of primary PLEC patients. Before conducting multicenter studies with larger samples in future, these nomograms were developed for simple usage and readily available prognostic tools may have potential value in promoting treatment decision-making and individualized prognosis prediction more effectively in clinical practice.

Data availability

Any reasonable requests for access to available data underlying the results reported in this article will be considered. Such proposals should be submitted to the corresponding author.

Abbreviations

• Computed tomography

Pulmonary lymphoepithelial carcinoma

Disease-free survival

Cancer-specific survival

Concordance index

Cytokeratin fragment antigen 21 − 1

EBV-encoded small non-polyadenylated RNAs

Epidermal growth factor receptor

Anaplastic lymphoma kinase

Kirsten rat sarcoma viral oncogene

Standard deviation

Interquartile ranges

Right upper lobe

Right middle lobe

Right lower lobe

Left upper lobe

Left lower lobe

Lymphadenopathy

Hazard ratio

Confidence interval

Area under the curve

Hu Y, Ren S, Liu Y, et al. Pulmonary lymphoepithelioma-like carcinoma: a mini-review. Onco Targets Ther. 2020;13(undefined):3921–9.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Liang Y, Wang L, Zhu Y, et al. Primary pulmonary lymphoepithelioma-like carcinoma: fifty-two patients with long-term follow-up. Cancer. 2012;118(19):4748–58. 10.10 02/cncr.274.52.

Article   CAS   PubMed   Google Scholar  

Qin Y, Gao G, Xie X Clinical features and prognosis of pulmonary lymphoepithelioma-like carcinoma: summary of eighty-five cases. Clin Lung Cancer., 2019;20(3): e329?e337

Bégin LR, Eskandari J, Joncas J et al. Epstein-Barr virus related lymphoepithelioma-like carcinoma of lung. J Surg Oncol, 1987, 36(4): 280–283.

He J, Shen J, Pan H, et al. Pulmonary lymphoepithelioma-like carcinoma: a surveillance, epidemiology, and end results database analysis. J Thorac Dis. 2015;7(12):2330–8.

PubMed   PubMed Central   Google Scholar  

Travis WD, Brambilla E, Muller-Hermelink HK, et al. Pathology and Genetics of tumors of the lung, Pleura, Thymus and heart. Volume 9. Lyon: IARC; 2004. pp. 457–65. 50.

Google Scholar  

Travis WD, Brambilla E, Burke AP, Marx A, Nicholson AG. Introduction to the 2015 World Health Organization Classification of Tumors of the lung, Pleura, Thymus, and heart. J Thorac Oncol. 2015;10(9):1240–2.

Article   PubMed   Google Scholar  

Tsao MS, Nicholson AG, Maleszewski JJ, Marx A, Travis WD. Introduction to 2021 WHO classification of thoracic tumors. J Thorac Oncol. 2022;17(1):e1–4.

Yang H, Lin Y, Liang Y. Treatment of Lung Carcinosarcoma and other Rare histologic subtypes of non-small cell Lung Cancer. Curr Treat Options Oncol. 2017;18(9):54. Published 2017 Aug 10.

Ettinger DS, Wood DE, Aisner DL, et al. NCCN guidelines Insights: Non-small Cell Lung Cancer, Version 2.2021. J Natl Compr Canc Netw. 2021;19(3):254–66. Published 2021 Mar 2.

Chen B, Zhang Y, Dai S, et al. Molecular characteristics of primary pulmonary lymphoepithelioma-like carcinoma based on integrated genomic analyses. Signal Transduct Target Ther. 2021;6(1):6. Published 2021 Jan 8.

Ball L, Vercesi V, Costantino F, Chandrapatham K, Pelosi P. Lung imaging: how to get better look inside the lung. Ann Transl Med. 2017;5(14):294.

Article   PubMed   PubMed Central   Google Scholar  

van Laar M, van Amsterdam WAC, van Lindert ASR, de Jong PA, Verhoeff JJC. Prognostic factors for overall survival of stage III non-small cell lung cancer patients on computed tomography: a systematic review and meta-analysis. Radiother Oncol. 2020;151:152–75.

Ooi GC, Ho JC, Khong PL, Wong MP, Lam WK, Tsang KW. Computed tomography characteristics of advanced primary pulmonary lymphoepithelioma-like carcinoma. Eur Radiol. 2003;13(3):522–6.

Ma H, Wu Y, Lin Y, Cai Q, Ma G, Liang Y. Computed tomography characteristics of primary pulmonary lymphoepithelioma-like carcinoma in 41 patients. Eur J Radiol. 2013;82(8):1343–6.

Chen B, Chen X, Zhou P, et al. Primary pulmonary lymphoepithelioma-like carcinoma: a rare type of lung cancer with a favorable outcome in comparison to squamous carcinoma. Respir Res. 2019;20(1):262. Published 2019 Nov 21.

Shen Y, Hu F, Zhang B, Li C, Zhang X, Han B. Clinicopathological characteristics with EGFR, ALK, ROS1 genetic alternation and prognostic analysis of primary lymphoepithelioma-like carcinoma. Transl Cancer Res. 2019;8(6):2350–6.

Jiang RR, Feng XL, Zhu WT, et al. A rare subtype of non-small cell lung Cancer: report of 159 Resected Pathological Stage I-IIIA Pulmonary Lymphoepithelioma-Like Carcinoma cases. Front Surg. 2021;8:757085. Published 2021 Oct 27.

Detterbeck FC, Boffa DJ, Kim AW, Tanoue LT. The Eighth Edition Lung Cancer Stage classification. Chest. 2017;151(1):193–203.

Camp RL, Dolled-Filhart M, Rimm DL. X-tile: a new bio-informatics tool for biomarker assessment and outcome-based cutpoint optimization. Clin Cancer Res. 2004;10(21):7252–9.

McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Meth. 1996;1(1):30–46.

Article   Google Scholar  

Yin WH, Lu B, Li N, Han L, Hou ZH, Wu RZ, Wu YJ, Niu HX, Jiang SL, Krazinski AW, Ebersberger U, Meinel FG, Schoepf UJ. Iterative reconstruction to preserve image quality and diagnostic accuracy at reduced radiation dose in coronary CT angiography: an intraindividual comparison, JACC Cardiovasc. Imag. 2013;6(12):1239–49.

Tay CK, Chua YC, Takano A, et al. Primary pulmonary lymphoepithelioma-like carcinoma in Singapore. Ann Thorac Med. 2018;13(1):30–5.

Su TP, Ho KC, Wang CW, et al. Prognostic value and clinical impact of pre-treatment FDG PET in Pulmonary Lymphoepithelioma-Like Carcinoma. Clin Nucl Med. 2019;44(2):e68–75.

Lei Y, Zhou J, Liu J, et al. The CT and PET/CT findings in primary pulmonary lymphoepithelioma-like carcinoma with pathological correlation: a study of 215 cases. Clin Radiol. 2022;77(3):e201–7.

Lin Z, Situ D, Chang X, et al. Surgical treatment for primary pulmonary lymphoepithelioma-like carcinoma. Interact Cardiovasc Thorac Surg. 2016;23(1):41–6.

Wang YN, Yao S, Wang CL, et al. Clinical significance of 4L Lymph Node Dissection in Left Lung Cancer. J Clin Oncol. 2018;36(29):2935–42.

Yu XQ, Yap ML, Cheng ES, et al. Evaluating prognostic factors for sex differences in Lung Cancer Survival: findings from a large Australian cohort. J Thorac Oncol. 2022;17(5):688–99.

Park SY, Lee JG, Kim J, Park Y, Lee SK, Bae MK, Lee CY, Kim DJ, Chung KY. Preoperative serum CYFRA 21 – 1 level as a prognostic factor in surgically treated adenocarcinoma of lung. Lung Cancer. 2013;79(2):156–60.

Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating pre-diction models. Med Decis Mak. 2006;26(6):565–74.

Download references

Acknowledgements

This work was financially supported by National Natural Science Foundation of China (Grant No. 82071873 and 81871353), Shanghai Municipal Commission of Science and Technology (Grant No. 22Y11911100), Shanghai Pujiang Program (Grant No. 22PJD069) and Shanghai Health Research Foundation for Talents (Grant No. 2022YQ060).

Author information

Authors and affiliations.

Department of Radiology, Shanghai Chest Hospital, Shanghai Jiao Tong University School of Medicine, No. 241 Huai-Hai West Road, Shanghai, 200030, P. R. China

Kai Nie, Lin Zhu, Yinan Chen & Hong Yu

Department of Pharmacology, University of Oxford, Oxford, OX1 3QT, UK

Yuxuan Zhang & John Parrington

You can also search for this author in PubMed   Google Scholar

Contributions

KN were involved in the literature search, figures, study design, data collection, data analysis, and writing. LZ, YXZ, and YNC were involved in literature search and data collection. J.P and HY were involved in study design and article review. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Hong Yu .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Ethics Committee of Shanghai Chest Hospital. Informed consent was waived because data were deidentified.

Consent for publication

Not applicable.

Competing interests

All remaining authors have declared no conflicts of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

12931_2024_2767_MOESM1_ESM.tif

Supplementary Material 1. Fig. S1 Kaplan Meier curve for CSS of total patients (a), the forest plot of factors obtained through multivariate COX regression analysis for CSS (b), the nomogram established for prediction of CSS (c), area under the curves at 3-year and 5-year were calculated to assess the prognostic accuracy for CSS (d), time dependent C index of nomogram of all PLEC patient (blue lines) and internally validated using a bootstrap resampling method (red lines) for CSS (e), calibration curves for 3 , 5 year CSS of nomogram predictions (f), decision curve analysis of nomogram for 3 year CSS (g) and 5 year CSS (h) of PLEC patients and Kaplan Meier curve for CSS based on the nomogram prediction (i).

Supplementary Material 2

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Nie, K., Zhu, L., Zhang, Y. et al. Development of a nomogram based on the clinicopathological and CT features to predict the survival of primary pulmonary lymphoepithelial carcinoma patients. Respir Res 25 , 144 (2024). https://doi.org/10.1186/s12931-024-02767-5

Download citation

Received : 23 August 2023

Accepted : 12 March 2024

Published : 29 March 2024

DOI : https://doi.org/10.1186/s12931-024-02767-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Lung cancer
• Prognostic factors

Respiratory Research

ISSN: 1465-993X

• General enquiries: [email protected]

Research on the Application of HHT Time-Frequency Analysis Method in Damage Analysis of Integrated Pipe Gallery

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.