social research analysis methods

2.2 Research Methods

Learning objectives.

By the end of this section, you should be able to:

Recall the 6 Steps of the Scientific Method
Differentiate between four kinds of research methods: surveys, field research, experiments, and secondary data analysis.
Explain the appropriateness of specific research approaches for specific topics.

Sociologists examine the social world, see a problem or interesting pattern, and set out to study it. They use research methods to design a study. Planning the research design is a key step in any sociological study. Sociologists generally choose from widely used methods of social investigation: primary source data collection such as survey, participant observation, ethnography, case study, unobtrusive observations, experiment, and secondary data analysis , or use of existing sources. Every research method comes with plusses and minuses, and the topic of study strongly influences which method or methods are put to use. When you are conducting research think about the best way to gather or obtain knowledge about your topic, think of yourself as an architect. An architect needs a blueprint to build a house, as a sociologist your blueprint is your research design including your data collection method.

When entering a particular social environment, a researcher must be careful. There are times to remain anonymous and times to be overt. There are times to conduct interviews and times to simply observe. Some participants need to be thoroughly informed; others should not know they are being observed. A researcher wouldn’t stroll into a crime-ridden neighborhood at midnight, calling out, “Any gang members around?”

Making sociologists’ presence invisible is not always realistic for other reasons. That option is not available to a researcher studying prison behaviors, early education, or the Ku Klux Klan. Researchers can’t just stroll into prisons, kindergarten classrooms, or Klan meetings and unobtrusively observe behaviors or attract attention. In situations like these, other methods are needed. Researchers choose methods that best suit their study topics, protect research participants or subjects, and that fit with their overall approaches to research.

As a research method, a survey collects data from subjects who respond to a series of questions about behaviors and opinions, often in the form of a questionnaire or an interview. The survey is one of the most widely used scientific research methods. The standard survey format allows individuals a level of anonymity in which they can express personal ideas.

At some point, most people in the United States respond to some type of survey. The 2020 U.S. Census is an excellent example of a large-scale survey intended to gather sociological data. Since 1790, United States has conducted a survey consisting of six questions to received demographical data pertaining to residents. The questions pertain to the demographics of the residents who live in the United States. Currently, the Census is received by residents in the United Stated and five territories and consists of 12 questions.

Not all surveys are considered sociological research, however, and many surveys people commonly encounter focus on identifying marketing needs and strategies rather than testing a hypothesis or contributing to social science knowledge. Questions such as, “How many hot dogs do you eat in a month?” or “Were the staff helpful?” are not usually designed as scientific research. The Nielsen Ratings determine the popularity of television programming through scientific market research. However, polls conducted by television programs such as American Idol or So You Think You Can Dance cannot be generalized, because they are administered to an unrepresentative population, a specific show’s audience. You might receive polls through your cell phones or emails, from grocery stores, restaurants, and retail stores. They often provide you incentives for completing the survey.

Sociologists conduct surveys under controlled conditions for specific purposes. Surveys gather different types of information from people. While surveys are not great at capturing the ways people really behave in social situations, they are a great method for discovering how people feel, think, and act—or at least how they say they feel, think, and act. Surveys can track preferences for presidential candidates or reported individual behaviors (such as sleeping, driving, or texting habits) or information such as employment status, income, and education levels.

A survey targets a specific population , people who are the focus of a study, such as college athletes, international students, or teenagers living with type 1 (juvenile-onset) diabetes. Most researchers choose to survey a small sector of the population, or a sample , a manageable number of subjects who represent a larger population. The success of a study depends on how well a population is represented by the sample. In a random sample , every person in a population has the same chance of being chosen for the study. As a result, a Gallup Poll, if conducted as a nationwide random sampling, should be able to provide an accurate estimate of public opinion whether it contacts 2,000 or 10,000 people.

After selecting subjects, the researcher develops a specific plan to ask questions and record responses. It is important to inform subjects of the nature and purpose of the survey up front. If they agree to participate, researchers thank subjects and offer them a chance to see the results of the study if they are interested. The researcher presents the subjects with an instrument, which is a means of gathering the information.

A common instrument is a questionnaire. Subjects often answer a series of closed-ended questions . The researcher might ask yes-or-no or multiple-choice questions, allowing subjects to choose possible responses to each question. This kind of questionnaire collects quantitative data —data in numerical form that can be counted and statistically analyzed. Just count up the number of “yes” and “no” responses or correct answers, and chart them into percentages.

Questionnaires can also ask more complex questions with more complex answers—beyond “yes,” “no,” or checkbox options. These types of inquiries use open-ended questions that require short essay responses. Participants willing to take the time to write those answers might convey personal religious beliefs, political views, goals, or morals. The answers are subjective and vary from person to person. How do you plan to use your college education?

Some topics that investigate internal thought processes are impossible to observe directly and are difficult to discuss honestly in a public forum. People are more likely to share honest answers if they can respond to questions anonymously. This type of personal explanation is qualitative data —conveyed through words. Qualitative information is harder to organize and tabulate. The researcher will end up with a wide range of responses, some of which may be surprising. The benefit of written opinions, though, is the wealth of in-depth material that they provide.

An interview is a one-on-one conversation between the researcher and the subject, and it is a way of conducting surveys on a topic. However, participants are free to respond as they wish, without being limited by predetermined choices. In the back-and-forth conversation of an interview, a researcher can ask for clarification, spend more time on a subtopic, or ask additional questions. In an interview, a subject will ideally feel free to open up and answer questions that are often complex. There are no right or wrong answers. The subject might not even know how to answer the questions honestly.

Questions such as “How does society’s view of alcohol consumption influence your decision whether or not to take your first sip of alcohol?” or “Did you feel that the divorce of your parents would put a social stigma on your family?” involve so many factors that the answers are difficult to categorize. A researcher needs to avoid steering or prompting the subject to respond in a specific way; otherwise, the results will prove to be unreliable. The researcher will also benefit from gaining a subject’s trust, from empathizing or commiserating with a subject, and from listening without judgment.

Surveys often collect both quantitative and qualitative data. For example, a researcher interviewing people who are incarcerated might receive quantitative data, such as demographics – race, age, sex, that can be analyzed statistically. For example, the researcher might discover that 20 percent of incarcerated people are above the age of 50. The researcher might also collect qualitative data, such as why people take advantage of educational opportunities during their sentence and other explanatory information.

The survey can be carried out online, over the phone, by mail, or face-to-face. When researchers collect data outside a laboratory, library, or workplace setting, they are conducting field research, which is our next topic.

Field Research

The work of sociology rarely happens in limited, confined spaces. Rather, sociologists go out into the world. They meet subjects where they live, work, and play. Field research refers to gathering primary data from a natural environment. To conduct field research, the sociologist must be willing to step into new environments and observe, participate, or experience those worlds. In field work, the sociologists, rather than the subjects, are the ones out of their element.

The researcher interacts with or observes people and gathers data along the way. The key point in field research is that it takes place in the subject’s natural environment, whether it’s a coffee shop or tribal village, a homeless shelter or the DMV, a hospital, airport, mall, or beach resort.

While field research often begins in a specific setting , the study’s purpose is to observe specific behaviors in that setting. Field work is optimal for observing how people think and behave. It seeks to understand why they behave that way. However, researchers may struggle to narrow down cause and effect when there are so many variables floating around in a natural environment. And while field research looks for correlation, its small sample size does not allow for establishing a causal relationship between two variables. Indeed, much of the data gathered in sociology do not identify a cause and effect but a correlation .

Sociology in the Real World

Beyoncé and lady gaga as sociological subjects.

Sociologists have studied Lady Gaga and Beyoncé and their impact on music, movies, social media, fan participation, and social equality. In their studies, researchers have used several research methods including secondary analysis, participant observation, and surveys from concert participants.

In their study, Click, Lee & Holiday (2013) interviewed 45 Lady Gaga fans who utilized social media to communicate with the artist. These fans viewed Lady Gaga as a mirror of themselves and a source of inspiration. Like her, they embrace not being a part of mainstream culture. Many of Lady Gaga’s fans are members of the LGBTQ community. They see the “song “Born This Way” as a rallying cry and answer her calls for “Paws Up” with a physical expression of solidarity—outstretched arms and ﬁngers bent and curled to resemble monster claws.”

Sascha Buchanan (2019) made use of participant observation to study the relationship between two fan groups, that of Beyoncé and that of Rihanna. She observed award shows sponsored by iHeartRadio, MTV EMA, and BET that pit one group against another as they competed for Best Fan Army, Biggest Fans, and FANdemonium. Buchanan argues that the media thus sustains a myth of rivalry between the two most commercially successful Black women vocal artists.

Participant Observation

In 2000, a comic writer named Rodney Rothman wanted an insider’s view of white-collar work. He slipped into the sterile, high-rise offices of a New York “dot com” agency. Every day for two weeks, he pretended to work there. His main purpose was simply to see whether anyone would notice him or challenge his presence. No one did. The receptionist greeted him. The employees smiled and said good morning. Rothman was accepted as part of the team. He even went so far as to claim a desk, inform the receptionist of his whereabouts, and attend a meeting. He published an article about his experience in The New Yorker called “My Fake Job” (2000). Later, he was discredited for allegedly fabricating some details of the story and The New Yorker issued an apology. However, Rothman’s entertaining article still offered fascinating descriptions of the inside workings of a “dot com” company and exemplified the lengths to which a writer, or a sociologist, will go to uncover material.

Rothman had conducted a form of study called participant observation , in which researchers join people and participate in a group’s routine activities for the purpose of observing them within that context. This method lets researchers experience a specific aspect of social life. A researcher might go to great lengths to get a firsthand look into a trend, institution, or behavior. A researcher might work as a waitress in a diner, experience homelessness for several weeks, or ride along with police officers as they patrol their regular beat. Often, these researchers try to blend in seamlessly with the population they study, and they may not disclose their true identity or purpose if they feel it would compromise the results of their research.

At the beginning of a field study, researchers might have a question: “What really goes on in the kitchen of the most popular diner on campus?” or “What is it like to be homeless?” Participant observation is a useful method if the researcher wants to explore a certain environment from the inside.

Field researchers simply want to observe and learn. In such a setting, the researcher will be alert and open minded to whatever happens, recording all observations accurately. Soon, as patterns emerge, questions will become more specific, observations will lead to hypotheses, and hypotheses will guide the researcher in analyzing data and generating results.

In a study of small towns in the United States conducted by sociological researchers John S. Lynd and Helen Merrell Lynd, the team altered their purpose as they gathered data. They initially planned to focus their study on the role of religion in U.S. towns. As they gathered observations, they realized that the effect of industrialization and urbanization was the more relevant topic of this social group. The Lynds did not change their methods, but they revised the purpose of their study.

This shaped the structure of Middletown: A Study in Modern American Culture , their published results (Lynd & Lynd, 1929).

The Lynds were upfront about their mission. The townspeople of Muncie, Indiana, knew why the researchers were in their midst. But some sociologists prefer not to alert people to their presence. The main advantage of covert participant observation is that it allows the researcher access to authentic, natural behaviors of a group’s members. The challenge, however, is gaining access to a setting without disrupting the pattern of others’ behavior. Becoming an inside member of a group, organization, or subculture takes time and effort. Researchers must pretend to be something they are not. The process could involve role playing, making contacts, networking, or applying for a job.

Once inside a group, some researchers spend months or even years pretending to be one of the people they are observing. However, as observers, they cannot get too involved. They must keep their purpose in mind and apply the sociological perspective. That way, they illuminate social patterns that are often unrecognized. Because information gathered during participant observation is mostly qualitative, rather than quantitative, the end results are often descriptive or interpretive. The researcher might present findings in an article or book and describe what he or she witnessed and experienced.

This type of research is what journalist Barbara Ehrenreich conducted for her book Nickel and Dimed . One day over lunch with her editor, Ehrenreich mentioned an idea. How can people exist on minimum-wage work? How do low-income workers get by? she wondered. Someone should do a study . To her surprise, her editor responded, Why don’t you do it?

That’s how Ehrenreich found herself joining the ranks of the working class. For several months, she left her comfortable home and lived and worked among people who lacked, for the most part, higher education and marketable job skills. Undercover, she applied for and worked minimum wage jobs as a waitress, a cleaning woman, a nursing home aide, and a retail chain employee. During her participant observation, she used only her income from those jobs to pay for food, clothing, transportation, and shelter.

She discovered the obvious, that it’s almost impossible to get by on minimum wage work. She also experienced and observed attitudes many middle and upper-class people never think about. She witnessed firsthand the treatment of working class employees. She saw the extreme measures people take to make ends meet and to survive. She described fellow employees who held two or three jobs, worked seven days a week, lived in cars, could not pay to treat chronic health conditions, got randomly fired, submitted to drug tests, and moved in and out of homeless shelters. She brought aspects of that life to light, describing difficult working conditions and the poor treatment that low-wage workers suffer.

The book she wrote upon her return to her real life as a well-paid writer, has been widely read and used in many college classrooms.

Ethnography

Ethnography is the immersion of the researcher in the natural setting of an entire social community to observe and experience their everyday life and culture. The heart of an ethnographic study focuses on how subjects view their own social standing and how they understand themselves in relation to a social group.

An ethnographic study might observe, for example, a small U.S. fishing town, an Inuit community, a village in Thailand, a Buddhist monastery, a private boarding school, or an amusement park. These places all have borders. People live, work, study, or vacation within those borders. People are there for a certain reason and therefore behave in certain ways and respect certain cultural norms. An ethnographer would commit to spending a determined amount of time studying every aspect of the chosen place, taking in as much as possible.

A sociologist studying a tribe in the Amazon might watch the way villagers go about their daily lives and then write a paper about it. To observe a spiritual retreat center, an ethnographer might sign up for a retreat and attend as a guest for an extended stay, observe and record data, and collate the material into results.

Institutional Ethnography

Institutional ethnography is an extension of basic ethnographic research principles that focuses intentionally on everyday concrete social relationships. Developed by Canadian sociologist Dorothy E. Smith (1990), institutional ethnography is often considered a feminist-inspired approach to social analysis and primarily considers women’s experiences within male- dominated societies and power structures. Smith’s work is seen to challenge sociology’s exclusion of women, both academically and in the study of women’s lives (Fenstermaker, n.d.).

Historically, social science research tended to objectify women and ignore their experiences except as viewed from the male perspective. Modern feminists note that describing women, and other marginalized groups, as subordinates helps those in authority maintain their own dominant positions (Social Sciences and Humanities Research Council of Canada n.d.). Smith’s three major works explored what she called “the conceptual practices of power” and are still considered seminal works in feminist theory and ethnography (Fensternmaker n.d.).

Sociological Research

The making of middletown: a study in modern u.s. culture.

In 1924, a young married couple named Robert and Helen Lynd undertook an unprecedented ethnography: to apply sociological methods to the study of one U.S. city in order to discover what “ordinary” people in the United States did and believed. Choosing Muncie, Indiana (population about 30,000) as their subject, they moved to the small town and lived there for eighteen months.

Ethnographers had been examining other cultures for decades—groups considered minorities or outsiders—like gangs, immigrants, and the poor. But no one had studied the so-called average American.

Recording interviews and using surveys to gather data, the Lynds objectively described what they observed. Researching existing sources, they compared Muncie in 1890 to the Muncie they observed in 1924. Most Muncie adults, they found, had grown up on farms but now lived in homes inside the city. As a result, the Lynds focused their study on the impact of industrialization and urbanization.

They observed that Muncie was divided into business and working class groups. They defined business class as dealing with abstract concepts and symbols, while working class people used tools to create concrete objects. The two classes led different lives with different goals and hopes. However, the Lynds observed, mass production offered both classes the same amenities. Like wealthy families, the working class was now able to own radios, cars, washing machines, telephones, vacuum cleaners, and refrigerators. This was an emerging material reality of the 1920s.

As the Lynds worked, they divided their manuscript into six chapters: Getting a Living, Making a Home, Training the Young, Using Leisure, Engaging in Religious Practices, and Engaging in Community Activities.

When the study was completed, the Lynds encountered a big problem. The Rockefeller Foundation, which had commissioned the book, claimed it was useless and refused to publish it. The Lynds asked if they could seek a publisher themselves.

Middletown: A Study in Modern American Culture was not only published in 1929 but also became an instant bestseller, a status unheard of for a sociological study. The book sold out six printings in its first year of publication, and has never gone out of print (Caplow, Hicks, & Wattenberg. 2000).

Nothing like it had ever been done before. Middletown was reviewed on the front page of the New York Times. Readers in the 1920s and 1930s identified with the citizens of Muncie, Indiana, but they were equally fascinated by the sociological methods and the use of scientific data to define ordinary people in the United States. The book was proof that social data was important—and interesting—to the U.S. public.

Sometimes a researcher wants to study one specific person or event. A case study is an in-depth analysis of a single event, situation, or individual. To conduct a case study, a researcher examines existing sources like documents and archival records, conducts interviews, engages in direct observation and even participant observation, if possible.

Researchers might use this method to study a single case of a foster child, drug lord, cancer patient, criminal, or rape victim. However, a major criticism of the case study as a method is that while offering depth on a topic, it does not provide enough evidence to form a generalized conclusion. In other words, it is difficult to make universal claims based on just one person, since one person does not verify a pattern. This is why most sociologists do not use case studies as a primary research method.

However, case studies are useful when the single case is unique. In these instances, a single case study can contribute tremendous insight. For example, a feral child, also called “wild child,” is one who grows up isolated from human beings. Feral children grow up without social contact and language, which are elements crucial to a “civilized” child’s development. These children mimic the behaviors and movements of animals, and often invent their own language. There are only about one hundred cases of “feral children” in the world.

As you may imagine, a feral child is a subject of great interest to researchers. Feral children provide unique information about child development because they have grown up outside of the parameters of “normal” growth and nurturing. And since there are very few feral children, the case study is the most appropriate method for researchers to use in studying the subject.

At age three, a Ukranian girl named Oxana Malaya suffered severe parental neglect. She lived in a shed with dogs, and she ate raw meat and scraps. Five years later, a neighbor called authorities and reported seeing a girl who ran on all fours, barking. Officials brought Oxana into society, where she was cared for and taught some human behaviors, but she never became fully socialized. She has been designated as unable to support herself and now lives in a mental institution (Grice 2011). Case studies like this offer a way for sociologists to collect data that may not be obtained by any other method.

Experiments

You have probably tested some of your own personal social theories. “If I study at night and review in the morning, I’ll improve my retention skills.” Or, “If I stop drinking soda, I’ll feel better.” Cause and effect. If this, then that. When you test the theory, your results either prove or disprove your hypothesis.

One way researchers test social theories is by conducting an experiment , meaning they investigate relationships to test a hypothesis—a scientific approach.

There are two main types of experiments: lab-based experiments and natural or field experiments. In a lab setting, the research can be controlled so that more data can be recorded in a limited amount of time. In a natural or field- based experiment, the time it takes to gather the data cannot be controlled but the information might be considered more accurate since it was collected without interference or intervention by the researcher.

As a research method, either type of sociological experiment is useful for testing if-then statements: if a particular thing happens (cause), then another particular thing will result (effect). To set up a lab-based experiment, sociologists create artificial situations that allow them to manipulate variables.

Classically, the sociologist selects a set of people with similar characteristics, such as age, class, race, or education. Those people are divided into two groups. One is the experimental group and the other is the control group. The experimental group is exposed to the independent variable(s) and the control group is not. To test the benefits of tutoring, for example, the sociologist might provide tutoring to the experimental group of students but not to the control group. Then both groups would be tested for differences in performance to see if tutoring had an effect on the experimental group of students. As you can imagine, in a case like this, the researcher would not want to jeopardize the accomplishments of either group of students, so the setting would be somewhat artificial. The test would not be for a grade reflected on their permanent record of a student, for example.

And if a researcher told the students they would be observed as part of a study on measuring the effectiveness of tutoring, the students might not behave naturally. This is called the Hawthorne effect —which occurs when people change their behavior because they know they are being watched as part of a study. The Hawthorne effect is unavoidable in some research studies because sociologists have to make the purpose of the study known. Subjects must be aware that they are being observed, and a certain amount of artificiality may result (Sonnenfeld 1985).

A real-life example will help illustrate the process. In 1971, Frances Heussenstamm, a sociology professor at California State University at Los Angeles, had a theory about police prejudice. To test her theory, she conducted research. She chose fifteen students from three ethnic backgrounds: Black, White, and Hispanic. She chose students who routinely drove to and from campus along Los Angeles freeway routes, and who had had perfect driving records for longer than a year.

Next, she placed a Black Panther bumper sticker on each car. That sticker, a representation of a social value, was the independent variable. In the 1970s, the Black Panthers were a revolutionary group actively fighting racism. Heussenstamm asked the students to follow their normal driving patterns. She wanted to see whether seeming support for the Black Panthers would change how these good drivers were treated by the police patrolling the highways. The dependent variable would be the number of traffic stops/citations.

The first arrest, for an incorrect lane change, was made two hours after the experiment began. One participant was pulled over three times in three days. He quit the study. After seventeen days, the fifteen drivers had collected a total of thirty-three traffic citations. The research was halted. The funding to pay traffic fines had run out, and so had the enthusiasm of the participants (Heussenstamm, 1971).

Perspectives in Social Research Methods and Analysis A Reader for Sociology

Howard Lune - Hunter College, USA
Enrique S. Pumar - The Catholic University of America
Ross Koppel - University of Pennsylvania, USA
Description

Offering accessible examples showcasing how sociologists conduct research

Comprising 22 reading from both academic journals and books, this text presents the best examples of research methodology in contemporary and classic sociology. The editors organized the readings according to the logic of a research project. Beginning with the research question, to design, data collection and analysis, to application in the world. Each section contains an introduction, serving as a mini-textbook that guides students through the steps of their own research. Discussion questions after each section further compel students to think about the lessons they have learned.

Key Features

Real research selections show research in action that underscores the relevance for students
Completely intact articles emphasize how the research is part of the whole sociological enterprise
By focusing on sociology, the works strengthen the role of the methods course in the major

This book is intended as either a core book or a secondary text, primarily for use in research methods courses in sociology.

The accompanying student study site http://www.sagepub.com/lunestudy includes:

Additional in-depth discussion questions per chapter
A section of additional questions for further review
Additional Web links as helpful resources
Journal articles related to the readings in the book
An "In the News" section that has a "real world" link to topics in the book

See what’s new to this edition by selecting the Features tab on this page. Should you need additional information or have questions regarding the HEOA information provided for this title, including what is new to this edition, please email [email protected] . Please include your name, contact information, and the name of the title for which you would like more information. For information on the HEOA, please go to http://ed.gov/policy/highered/leg/hea08/index.html .

For assistance with your order: Please email us at [email protected] or connect with your SAGE representative.

SAGE 2455 Teller Road Thousand Oaks, CA 91320 www.sagepub.com

This text does not fit well with the way the course is structured and the level of ability of my students.

Sample Materials & Chapters

Section 1: Where to Begin

Section 2: Research Design

For instructors

Select a purchasing option, related products.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

13 Qualitative analysis

Qualitative analysis is the analysis of qualitative data such as text data from interview transcripts. Unlike quantitative analysis, which is statistics driven and largely independent of the researcher, qualitative analysis is heavily dependent on the researcher’s analytic and integrative skills and personal knowledge of the social context where the data is collected. The emphasis in qualitative analysis is ‘sense making’ or understanding a phenomenon, rather than predicting or explaining. A creative and investigative mindset is needed for qualitative analysis, based on an ethically enlightened and participant-in-context attitude, and a set of analytic strategies. This chapter provides a brief overview of some of these qualitative analysis strategies. Interested readers are referred to more authoritative and detailed references such as Miles and Huberman’s (1984) [1] seminal book on this topic.

Grounded theory

How can you analyse a vast set of qualitative data acquired through participant observation, in-depth interviews, focus groups, narratives of audio/video recordings, or secondary documents? One of these techniques for analysing text data is grounded theory —an inductive technique of interpreting recorded data about a social phenomenon to build theories about that phenomenon. The technique was developed by Glaser and Strauss (1967) [2] in their method of constant comparative analysis of grounded theory research, and further refined by Strauss and Corbin (1990) [3] to further illustrate specific coding techniques—a process of classifying and categorising text data segments into a set of codes (concepts), categories (constructs), and relationships. The interpretations are ‘grounded in’ (or based on) observed empirical data, hence the name. To ensure that the theory is based solely on observed evidence, the grounded theory approach requires that researchers suspend any pre-existing theoretical expectations or biases before data analysis, and let the data dictate the formulation of the theory.

Strauss and Corbin (1998) describe three coding techniques for analysing text data: open, axial, and selective. Open coding is a process aimed at identifying concepts or key ideas that are hidden within textual data, which are potentially related to the phenomenon of interest. The researcher examines the raw textual data line by line to identify discrete events, incidents, ideas, actions, perceptions, and interactions of relevance that are coded as concepts (hence called in vivo codes ). Each concept is linked to specific portions of the text (coding unit) for later validation. Some concepts may be simple, clear, and unambiguous, while others may be complex, ambiguous, and viewed differently by different participants. The coding unit may vary with the concepts being extracted. Simple concepts such as ‘organisational size’ may include just a few words of text, while complex ones such as ‘organizational mission’ may span several pages. Concepts can be named using the researcher’s own naming convention, or standardised labels taken from the research literature. Once a basic set of concepts are identified, these concepts can then be used to code the remainder of the data, while simultaneously looking for new concepts and refining old concepts. While coding, it is important to identify the recognisable characteristics of each concept, such as its size, colour, or level—e.g., high or low—so that similar concepts can be grouped together later . This coding technique is called ‘open’ because the researcher is open to and actively seeking new concepts relevant to the phenomenon of interest.

Next, similar concepts are grouped into higher order categories . While concepts may be context-specific, categories tend to be broad and generalisable, and ultimately evolve into constructs in a grounded theory. Categories are needed to reduce the amount of concepts the researcher must work with and to build a ‘big picture’ of the issues salient to understanding a social phenomenon. Categorisation can be done in phases, by combining concepts into subcategories, and then subcategories into higher order categories. Constructs from the existing literature can be used to name these categories, particularly if the goal of the research is to extend current theories. However, caution must be taken while using existing constructs, as such constructs may bring with them commonly held beliefs and biases. For each category, its characteristics (or properties) and the dimensions of each characteristic should be identified. The dimension represents a value of a characteristic along a continuum. For example, a ‘communication media’ category may have a characteristic called ‘speed’, which can be dimensionalised as fast, medium, or slow . Such categorisation helps differentiate between different kinds of communication media, and enables researchers to identify patterns in the data, such as which communication media is used for which types of tasks.

The second phase of grounded theory is axial coding , where the categories and subcategories are assembled into causal relationships or hypotheses that can tentatively explain the phenomenon of interest. Although distinct from open coding, axial coding can be performed simultaneously with open coding. The relationships between categories may be clearly evident in the data, or may be more subtle and implicit. In the latter instance, researchers may use a coding scheme (often called a ‘coding paradigm’, but different from the paradigms discussed in Chapter 3) to understand which categories represent conditions (the circumstances in which the phenomenon is embedded), actions/interactions (the responses of individuals to events under these conditions), and consequences (the outcomes of actions/interactions). As conditions, actions/interactions, and consequences are identified, theoretical propositions start to emerge, and researchers can start explaining why a phenomenon occurs, under what conditions, and with what consequences.

The third and final phase of grounded theory is selective coding , which involves identifying a central category or a core variable, and systematically and logically relating this central category to other categories. The central category can evolve from existing categories or can be a higher order category that subsumes previously coded categories. New data is selectively sampled to validate the central category, and its relationships to other categories—i.e., the tentative theory. Selective coding limits the range of analysis, and makes it move fast. At the same time, the coder must watch out for other categories that may emerge from the new data that could be related to the phenomenon of interest (open coding), which may lead to further refinement of the initial theory. Hence, open, axial, and selective coding may proceed simultaneously. Coding of new data and theory refinement continues until theoretical saturation is reached—i.e., when additional data does not yield any marginal change in the core categories or the relationships.

The ‘constant comparison’ process implies continuous rearrangement, aggregation, and refinement of categories, relationships, and interpretations based on increasing depth of understanding, and an iterative interplay of four stages of activities: comparing incidents/texts assigned to each category to validate the category), integrating categories and their properties, delimiting the theory by focusing on the core concepts and ignoring less relevant concepts, and writing theory using techniques like memoing, storylining, and diagramming. Having a central category does not necessarily mean that all other categories can be integrated nicely around it. In order to identify key categories that are conditions, action/interactions, and consequences of the core category, Strauss and Corbin (1990) recommend several integration techniques, such as storylining, memoing, or concept mapping, which are discussed here. In storylining , categories and relationships are used to explicate and/or refine a story of the observed phenomenon. Memos are theorised write-ups of ideas about substantive concepts and their theoretically coded relationships as they evolve during ground theory analysis, and are important tools to keep track of and refine ideas that develop during the analysis. Memoing is the process of using these memos to discover patterns and relationships between categories using two-by-two tables, diagrams, or figures, or other illustrative displays. Concept mapping is a graphical representation of concepts and relationships between those concepts—e.g., using boxes and arrows. The major concepts are typically laid out on one or more sheets of paper, blackboards, or using graphical software programs, linked to each other using arrows, and readjusted to best fit the observed data.

After a grounded theory is generated, it must be refined for internal consistency and logic. Researchers must ensure that the central construct has the stated characteristics and dimensions, and if not, the data analysis may be repeated. Researcher must then ensure that the characteristics and dimensions of all categories show variation. For example, if behaviour frequency is one such category, then the data must provide evidence of both frequent performers and infrequent performers of the focal behaviour. Finally, the theory must be validated by comparing it with raw data. If the theory contradicts with observed evidence, the coding process may need to be repeated to reconcile such contradictions or unexplained variations.

Content analysis

Content analysis is the systematic analysis of the content of a text—e.g., who says what, to whom, why, and to what extent and with what effect—in a quantitative or qualitative manner. Content analysis is typically conducted as follows. First, when there are many texts to analyse—e.g., newspaper stories, financial reports, blog postings, online reviews, etc.—the researcher begins by sampling a selected set of texts from the population of texts for analysis. This process is not random, but instead, texts that have more pertinent content should be chosen selectively. Second, the researcher identifies and applies rules to divide each text into segments or ‘chunks’ that can be treated as separate units of analysis. This process is called unitising . For example, assumptions, effects, enablers, and barriers in texts may constitute such units. Third, the researcher constructs and applies one or more concepts to each unitised text segment in a process called coding . For coding purposes, a coding scheme is used based on the themes the researcher is searching for or uncovers as they classify the text. Finally, the coded data is analysed, often both quantitatively and qualitatively, to determine which themes occur most frequently, in what contexts, and how they are related to each other.

A simple type of content analysis is sentiment analysis —a technique used to capture people’s opinion or attitude toward an object, person, or phenomenon. Reading online messages about a political candidate posted on an online forum and classifying each message as positive, negative, or neutral is an example of such an analysis. In this case, each message represents one unit of analysis. This analysis will help identify whether the sample as a whole is positively or negatively disposed, or neutral towards that candidate. Examining the content of online reviews in a similar manner is another example. Though this analysis can be done manually, for very large datasets—e.g., millions of text records—natural language processing and text analytics based software programs are available to automate the coding process, and maintain a record of how people’s sentiments fluctuate with time.

A frequent criticism of content analysis is that it lacks a set of systematic procedures that would allow the analysis to be replicated by other researchers. Schilling (2006) [4] addressed this criticism by organising different content analytic procedures into a spiral model. This model consists of five levels or phases in interpreting text: convert recorded tapes into raw text data or transcripts for content analysis, convert raw data into condensed protocols, convert condensed protocols into a preliminary category system, use the preliminary category system to generate coded protocols, and analyse coded protocols to generate interpretations about the phenomenon of interest.

Content analysis has several limitations. First, the coding process is restricted to the information available in text form. For instance, if a researcher is interested in studying people’s views on capital punishment, but no such archive of text documents is available, then the analysis cannot be done. Second, sampling must be done carefully to avoid sampling bias. For instance, if your population is the published research literature on a given topic, then you have systematically omitted unpublished research or the most recent work that is yet to be published.

Hermeneutic analysis

Hermeneutic analysis is a special type of content analysis where the researcher tries to ‘interpret’ the subjective meaning of a given text within its sociohistoric context. Unlike grounded theory or content analysis—which ignores the context and meaning of text documents during the coding process—hermeneutic analysis is a truly interpretive technique for analysing qualitative data. This method assumes that written texts narrate an author’s experience within a sociohistoric context, and should be interpreted as such within that context. Therefore, the researcher continually iterates between singular interpretation of the text (the part) and a holistic understanding of the context (the whole) to develop a fuller understanding of the phenomenon in its situated context, which German philosopher Martin Heidegger called the hermeneutic circle . The word hermeneutic (singular) refers to one particular method or strand of interpretation.

More generally, hermeneutics is the study of interpretation and the theory and practice of interpretation. Derived from religious studies and linguistics, traditional hermeneutics—such as biblical hermeneutics —refers to the interpretation of written texts, especially in the areas of literature, religion and law—such as the Bible. In the twentieth century, Heidegger suggested that a more direct, non-mediated, and authentic way of understanding social reality is to experience it, rather than simply observe it, and proposed philosophical hermeneutics , where the focus shifted from interpretation to existential understanding. Heidegger argued that texts are the means by which readers can not only read about an author’s experience, but also relive the author’s experiences. Contemporary or modern hermeneutics, developed by Heidegger’s students such as Hans-Georg Gadamer, further examined the limits of written texts for communicating social experiences, and went on to propose a framework of the interpretive process, encompassing all forms of communication, including written, verbal, and non-verbal, and exploring issues that restrict the communicative ability of written texts, such as presuppositions, language structures (e.g., grammar, syntax, etc.), and semiotics—the study of written signs such as symbolism, metaphor, analogy, and sarcasm. The term hermeneutics is sometimes used interchangeably and inaccurately with exegesis , which refers to the interpretation or critical explanation of written text only, and especially religious texts.

Finally, standard software programs, such as ATLAS.ti.5, NVivo, and QDA Miner, can be used to automate coding processes in qualitative research methods. These programs can quickly and efficiently organise, search, sort, and process large volumes of text data using user-defined rules. To guide such automated analysis, a coding schema should be created, specifying the keywords or codes to search for in the text, based on an initial manual examination of sample text data. The schema can be arranged in a hierarchical manner to organise codes into higher-order codes or constructs. The coding schema should be validated using a different sample of texts for accuracy and adequacy. However, if the coding schema is biased or incorrect, the resulting analysis of the entire population of texts may be flawed and non-interpretable. However, software programs cannot decipher the meaning behind certain words or phrases or the context within which these words or phrases are used—such sarcasm or metaphors—which may lead to significant misinterpretation in large scale qualitative analysis.

Miles, M. B., & Huberman, A. M. (1984). Qualitative data analysis: A sourcebook of new methods . Newbury Park, CA: Sage Publications. ↵
Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research . New York: Aldine Pub Co. ↵
Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques , Beverly Hills: Sage Publications. ↵
Schiling, J. (2006). On the pragmatics of qualitative assessment: Designing the process for content analysis. European Journal of Psychological Assessment , 22(1), 28–37. ↵

Social Science Research: Principles, Methods and Practices (Revised edition) Copyright © 2019 by Anol Bhattacherjee is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base

Methodology

Research Methods | Definitions, Types, Examples

Research methods are specific procedures for collecting and analyzing data. Developing your research methods is an integral part of your research design . When planning your methods, there are two key decisions you will make.

First, decide how you will collect data . Your methods depend on what type of data you need to answer your research question :

Qualitative vs. quantitative : Will your data take the form of words or numbers?
Primary vs. secondary : Will you collect original data yourself, or will you use data that has already been collected by someone else?
Descriptive vs. experimental : Will you take measurements of something as it is, or will you perform an experiment?

Second, decide how you will analyze the data .

For quantitative data, you can use statistical analysis methods to test relationships between variables.
For qualitative data, you can use methods such as thematic analysis to interpret patterns and meanings in the data.

Methods for collecting data, examples of data collection methods, methods for analyzing data, examples of data analysis methods, other interesting articles, frequently asked questions about research methods.

Data is the information that you collect for the purposes of answering your research question . The type of data you need depends on the aims of your research.

Qualitative vs. quantitative data

Your choice of qualitative or quantitative data collection depends on the type of knowledge you want to develop.

For questions about ideas, experiences and meanings, or to study something that can’t be described numerically, collect qualitative data .

If you want to develop a more mechanistic understanding of a topic, or your research involves hypothesis testing , collect quantitative data .

You can also take a mixed methods approach , where you use both qualitative and quantitative research methods.

Primary vs. secondary research

Primary research is any original data that you collect yourself for the purposes of answering your research question (e.g. through surveys , observations and experiments ). Secondary research is data that has already been collected by other researchers (e.g. in a government census or previous scientific studies).

If you are exploring a novel research question, you’ll probably need to collect primary data . But if you want to synthesize existing knowledge, analyze historical trends, or identify patterns on a large scale, secondary data might be a better choice.

Descriptive vs. experimental data

In descriptive research , you collect data about your study subject without intervening. The validity of your research will depend on your sampling method .

In experimental research , you systematically intervene in a process and measure the outcome. The validity of your research will depend on your experimental design .

To conduct an experiment, you need to be able to vary your independent variable , precisely measure your dependent variable, and control for confounding variables . If it’s practically and ethically possible, this method is the best choice for answering questions about cause and effect.

Prevent plagiarism. Run a free check.

Your data analysis methods will depend on the type of data you collect and how you prepare it for analysis.

Data can often be analyzed both quantitatively and qualitatively. For example, survey responses could be analyzed qualitatively by studying the meanings of responses or quantitatively by studying the frequencies of responses.

Qualitative analysis methods

Qualitative analysis is used to understand words, ideas, and experiences. You can use it to interpret data that was collected:

From open-ended surveys and interviews , literature reviews , case studies , ethnographies , and other sources that use text rather than numbers.
Using non-probability sampling methods .

Qualitative analysis tends to be quite flexible and relies on the researcher’s judgement, so you have to reflect carefully on your choices and assumptions and be careful to avoid research bias .

Quantitative analysis methods

Quantitative analysis uses numbers and statistics to understand frequencies, averages and correlations (in descriptive studies) or cause-and-effect relationships (in experiments).

You can use quantitative analysis to interpret data that was collected either:

During an experiment .
Using probability sampling methods .

Because the data is collected and analyzed in a statistically valid way, the results of quantitative analysis can be easily standardized and shared among researchers.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

Chi square test of independence
Statistical power
Descriptive statistics
Degrees of freedom
Pearson correlation
Null hypothesis
Double-blind study
Case-control study
Research ethics
Data collection
Hypothesis testing
Structured interviews

Research bias

Hawthorne effect
Unconscious bias
Recall bias
Halo effect
Self-serving bias
Information bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

A sample is a subset of individuals from a larger population . Sampling means selecting the group that you will actually collect data from in your research. For example, if you are researching the opinions of students in your university, you could survey a sample of 100 students.

In statistics, sampling allows you to test a hypothesis about the characteristics of a population.

The research methods you use depend on the type of data you need to answer your research question .

If you want to measure something or test a hypothesis , use quantitative methods . If you want to explore ideas, thoughts and meanings, use qualitative methods .
If you want to analyze a large amount of readily-available data, use secondary data. If you want data specific to your purposes with control over how it is generated, collect primary data.
If you want to establish cause-and-effect relationships between variables , use experimental methods. If you want to understand the characteristics of a research subject, use descriptive methods.

Methodology refers to the overarching strategy and rationale of your research project . It involves studying the methods used in your field and the theories or principles behind them, in order to develop an approach that matches your objectives.

Methods are the specific tools and procedures you use to collect and analyze data (for example, experiments, surveys , and statistical tests ).

In shorter scientific papers, where the aim is to report the findings of a specific study, you might simply describe what you did in a methods section .

In a longer or more complex research project, such as a thesis or dissertation , you will probably include a methodology section , where you explain your approach to answering the research questions and cite relevant sources to support your choice of methods.

Is this article helpful?

Other students also liked, writing strong research questions | criteria & examples.

What Is a Research Design | Types, Guide & Examples
Data Collection | Definition, Methods & Examples

Unlimited Academic AI-Proofreading

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Skip to main content
Skip to primary sidebar
Skip to footer
QuestionPro

Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case NPS+ Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research

Social Research – Definition, Types and Methods

Social Research: Definition

Social Research is a method used by social scientists and researchers to learn about people and societies so that they can design products/services that cater to various needs of the people. Different socio-economic groups belonging to different parts of a county think differently. Various aspects of human behavior need to be addressed to understand their thoughts and feedback about the social world, which can be done using Social Research. Any topic can trigger social research – new feature, new market trend or an upgrade in old technology.

Select your respondents

Social Research is conducted by following a systematic plan of action which includes qualitative and quantitative observation methods.

Qualitative methods rely on direct communication with members of a market, observation, text analysis. The results of this method are focused more on being accurate rather than generalizing to the entire population.
Quantitative methods use statistical analysis techniques to evaluate data collected via surveys, polls or questionnaires.

LEARN ABOUT: Research Process Steps

Social Research contains elements of both these methods to analyze a range of social occurrences such as an investigation of historical sites, census of the country, detailed analysis of research conducted to understand reasons for increased reports of molestation in the country etc.

A survey to monitor happiness in a respondent population is one of the most widely used applications of social research. The happiness survey template can be used by researchers an organizations to gauge how happy a respondent is and the things that can be done to increase happiness in that respondent.

Learn more: Public Library Survey Questions + Sample Questionnaire Template

Types of Social Research

There are four main types of Social Research: Qualitative and Quantitative Research, Primary and Secondary Research.

Qualitative Research: Qualitative Research is defined as a method to collect data via open-ended and conversational discussions, There are five main qualitative research methods- ethnographic research, focus groups, one-on-one online interview, content analysis and case study research. Usually, participants are not taken out of their ecosystem for qualitative data collection to gather information in real-time which helps in building trust. Researchers depend on multiple methods to gather qualitative data for complex issues.

Quantitative Research: Quantitative Research is an extremely informative source of data collection conducted via mediums such as surveys, polls, and questionnaires. The gathered data can be analyzed to conclude numerical or statistical results. There are four distinct quantitative research methods: survey research , correlational research , causal research and experimental research . This research is carried out on a sample that is representative of the target market usually using close-ended questions and data is presented in tables, charts, graphs etc.

For example, A survey can be conducted to understand Climate change awareness among the general population. Such a survey will give in-depth information about people’s perception about climate change and also the behaviors that impact positive behavior. Such a questionnaire will enable the researcher to understand what needs to be done to create more awareness among the public.

Learn More: Climate Change Awareness Survey Template

Primary Research: Primary Research is conducted by the researchers themselves. There are a list of questions that a researcher intends to ask which need to be customized according to the target market. These questions are sent to the respondents via surveys, polls or questionnaires so that analyzing them becomes convenient for the researcher. Since data is collected first-hand, it’s highly accurate according to the requirement of research.

For example: There are tens of thousands of deaths and injuries related to gun violence in the United States. We keep hearing about people carrying weapons attacking general public in the news. There is quite a debate in the American public as to understand if possession of guns is the cause to this. Institutions related to public health or governmental organizations are carrying out studies to find the cause. A lot of policies are also influenced by the opinion of the general population and gun control policies are no different. Hence a gun control questionnaire can be carried out to gather data to understand what people think about gun violence, gun control, factors and effects of possession of firearms. Such a survey can help these institutions to make valid reforms on the basis of the data gathered.

Learn more: Wi-Fi Security Survey Questions + Sample Questionnaire Template

Secondary Research: Secondary Research is a method where information has already been collected by research organizations or marketers. Newspapers, online communities, reports, audio-visual evidence etc. fall under the category of secondary data. After identifying the topic of research and research sources, a researcher can collect existing information available from the noted sources. They can then combine all the information to compare and analyze it to derive conclusions.

LEARN ABOUT: Qualitative Research Questions and Questionnaires

Social Research Methods

Surveys: A survey is conducted by sending a set of pre-decided questions to a sample of individuals from a target market. This will lead to a collection of information and feedback from individuals that belong to various backgrounds, ethnicities, age-groups etc. Surveys can be conducted via online and offline mediums. Due to the improvement in technological mediums and their reach, online mediums have flourished and there is an increase in the number of people depending on online survey software to conduct regular surveys and polls.

There are various types of social research surveys: Longitudinal , Cross-sectional , Correlational Research . Longitudinal and Cross-sectional social research surveys are observational methods while Correlational is a non-experimental research method. Longitudinal social research surveys are conducted with the same sample over a course of time while Cross-sectional surveys are conducted with different samples.

For example: It has been observed in recent times, that there is an increase in the number of divorces, or failed relationships. The number of couples visiting marriage counselors or psychiatrists is increasing. Sometimes it gets tricky to understand what is the cause for a relationship falling apart. A screening process to understand an overview of the relationship can be an easy method. A marriage counselor can use a relationship survey to understand the chemistry in a relationship, the factors that influence the health of a relationship, the challenges faced in a relationship and expectations in a relationship. Such a survey can be very useful to deduce various findings in a patient and treatment can be done accordingly.

Another example for the use of surveys can be to gather information on the awareness of disasters and disaster management programs. A lot of institutions like the UN or the local disaster management team try to keep their communities prepared for disasters. Possessing knowledge about this is crucial in disaster prone areas and is a good type of knowledge that can help everyone. In such a case, a survey can enable these institutions to understand what are the areas that can be promoted more and what regions need what kind of training. Hence a disaster management survey can be conducted to understand public’s knowledge about the impact of disasters on communities, and the measures they undertake to respond to disasters and how can the risk be reduced.

Learn more: NBA Survey Questions + Sample Questionnaire Template

Experiments: An experimental research is conducted by researchers to observe the change in one variable on another, i.e. to establish the cause and effects of a variable. In experiments, there is a theory which needs to be proved or disproved by careful observation and analysis. An efficient experiment will be successful in building a cause-effect relationship while proving, rejecting or disproving a theory. Laboratory and field experiments are preferred by researchers.

Interviews: The technique of garnering opinions and feedback by asking selected questions face-to-face, via telephone or online mediums is called interview research. There are formal and informal interviews – formal interviews are the ones which are organized by the researcher with structured open-ended and closed-ended questions and format while informal interviews are the ones which are more of conversations with the participants and are extremely flexible to collect as much information as possible.

LEARN ABOUT: 12 Best Tools for Researchers

Examples of interviews in social research are sociological studies that are conducted to understand how religious people are. To this effect, a Church survey can be used by a pastor or priest to understand from the laity the reasons they attend Church and if it meets their spiritual needs.

Observation: In observational research , a researcher is expected to be involved in the daily life of all the participants to understand their routine, their decision-making skills, their capability to handle pressure and their overall likes and dislikes. These factors and recorded and careful observations are made to decide factors such as whether a change in law will impact their lifestyle or whether a new feature will be accepted by individuals.

Learn more:

Quantitative Observation

Qualitative Observation

MORE LIKE THIS

Top 17 UX Research Software for UX Design in 2024

Apr 5, 2024

Healthcare Staff Burnout: What it Is + How To Manage It

Apr 4, 2024

Top 15 Employee Retention Software in 2024

Top 10 Employee Development Software for Talent Growth

Apr 3, 2024

Social Research Methods

Definition of social research methods.

Social research methods are the tools that allow us to ask questions and find answers about the way people live. Imagine being a social detective, using these tools to solve the mysteries of human behavior, relationships, and communities. It’s similar to putting together a puzzle. As you gather more pieces (information), the picture of how we interact and why becomes clearer.

Another definition of social research methods is the strategies that provide us a lens to examine and understand the dynamics of society. Think of it as having a special pair of glasses that help you see the hidden connections and patterns in the everyday actions of people. With these glasses on, you can find answers to big questions, like why certain areas have more crime or what makes people happy with their jobs.

Types of Social Research Methods

There’s a range of techniques to explore various social questions. Each method is like a different detective tool, suited for certain kinds of clues.

Surveys: You distribute a list of questions to many individuals to quickly gather standardized responses on a topic.
Interviews: These are one-on-one conversations that provide in-depth understanding of someone’s feelings, thoughts, and experiences.
Observations: Observations involve watching people and settings without interference to notice behaviors and interactions.
Experiments: In this method, researchers manipulate certain conditions and observe outcomes to establish cause-and-effect relationships.
Content Analysis: This involves systematically examining text or media content to identify patterns, themes, or biases.

Examples of Social Research Methods

Here are some real-life applications of these methods to help illustrate how they work:

Reading through a celebrity’s Twitter feed: This is content analysis because you delve into their posts to uncover recurring themes and the interests of this person, revealing how public figures influence their followers.
Standing in the corner of a cafeteria: By watching how individuals interact and choose where to sit, you’re engaging in observation. This can help you map out social networks and how spaces influence social behavior .
Asking a bunch of high school students: Using a survey, you collect information on their studying habits. This lets you measure patterns and possibly improve educational strategies.
Chatting with a skateboarder: This interview can unlock detailed knowledge of the skateboarding subculture and attitudes toward risk-taking in sports.
Seeing if people are more likely to recycle: By conducting an experiment and providing more recycle bins, you can identify what actions influence environmental behavior.

Why is it Important?

Social research methods are key for more than just academics. They help us solve everyday problems, like finding ways to decrease bullying in schools or understanding how to encourage people to lead healthier lives. With research, a community could figure out the best steps to create a park that makes everyone happy, or a business could learn why some of its workers seem less motivated and how to fix that.

On a more personal level, understanding social research can help you make better decisions because it teaches you to look at information critically. You become aware of how your actions and the actions of those around you can influence society as a whole, from small interactions to significant social movements .

Origin of Social Research Methods

In the 19th century, sociologists like Comte and Durkheim pioneered the field of social research by advocating for systematic and scientific approaches to studying society, much as we might investigate the natural world.

Controversies in Social Research

Ethical dilemmas, such as protecting individuals’ confidentiality, avoiding bias, and establishing causality , are significant challenges in social research that require careful consideration and adherence to established guidelines and standards.

Social research methods provide us with a framework to investigate the complexities of human societies. By utilizing various techniques such as surveys, interviews, observations, experiments, and content analysis, we gain insights into human behavior and social interactions that guide improvements in our communities and policies. These methods empower us to seek truths and develop a deeper understanding of the world around us.

Social Data Analysis

Mikaila Mariel Lemonik Arthur, Rhode Island College

Roger Clark, Rhode Island College

Last Update: 2023

Publisher: Rhode Island College Digital Publishing

Language: English

Formats Available

Conditions of use.

Learn more about reviews.

Reviewed by Alice Cheng, Associate Professor, North Carolina State University on 12/19/23

Social Data Analysis: A Comprehensive Guide" truly lives up to its title by offering a comprehensive exploration of both quantitative and qualitative data analysis in the realm of social research. The book provides an in-depth understanding of the... read more

Comprehensiveness rating: 4 see less

The comprehensiveness of the book is evident in several key aspects:

Coverage of Quantitative and Qualitative Methods:

The book effectively covers both quantitative and qualitative data analysis, acknowledging the importance of a balanced approach in social research. Readers benefit from a holistic understanding of various analytical methods, allowing them to choose the most suitable approach for their research questions. Focus on SPSS for Quantitative Analysis:

The dedicated section on quantitative data analysis with SPSS demonstrates the book's commitment to providing practical guidance. Readers are taken through the nuances of using SPSS, from basic functions to more advanced analysis, enhancing their proficiency in a widely used statistical software. Real-World Application Using GSS Data:

The integration of data from the 2021 General Social Survey (GSS) and the modified GSS Codebook adds a practical dimension to the book. Readers have the opportunity to apply their learning to real-world scenarios, fostering a deeper understanding of social data analysis in action. Consideration of Ethical Practices:

The book's mention of survey weights and their exclusion from the learning dataset reflects a commitment to ethical data analysis practices. This attention to ethical considerations enhances the comprehensiveness of the book by addressing important aspects of responsible research. Supplementary Resources and Glossary:

The inclusion of a glossary ensures that readers, especially those new to the field, can easily grasp the terminology used. The availability of supplementary resources, such as a modified GSS Codebook, further supports readers in applying their knowledge beyond theoretical discussions. Recognition of Alternative Tools:

Acknowledging the existence of alternative tools, such as R, demonstrates the book's awareness of the diversity in data analysis approaches. While focusing on SPSS, the book encourages readers to explore other options, contributing to a more nuanced and well-rounded education in social data analysis. Overall, the book's comprehensiveness lies not only in its coverage of various data analysis methods but also in its commitment to providing practical, ethical, and diverse perspectives on social data analysis. It serves as an inclusive and accessible guide for readers at different levels of expertise.

Content Accuracy rating: 4

"Social Data Analysis: A Comprehensive Guide" maintains a commendable level of accuracy throughout its content. The authors demonstrate a meticulous approach to presenting information, ensuring that concepts are explained with precision and clarity. The accuracy is particularly notable in the sections covering quantitative data analysis with SPSS, where step-by-step instructions are provided for readers to follow, minimizing the risk of misinterpretation.

The use of real-world examples from the 2021 General Social Survey enhances the book's accuracy by grounding theoretical discussions in practical applications. The modified GSS Codebook is a thoughtful addition, contributing to the accuracy of the learning experience by providing a clear reference for variables used in the examples.

The authors' acknowledgment of the limitation regarding survey weights in the learning dataset reflects a commitment to transparency and ethical research practices. While the book focuses on a specific statistical software (SPSS), it accurately recognizes alternative tools like R, allowing readers to make informed decisions based on their preferences and requirements.

The glossary aids in maintaining accuracy by providing clear definitions of key terms, ensuring that readers have a precise understanding of the terminology used. Additionally, the reference to external resources, such as IBM's list of resellers and related guides from Kent State, contributes to the accuracy of the book by directing readers to authoritative sources for further information.

In conclusion, "Social Data Analysis: A Comprehensive Guide" upholds a high level of accuracy, presenting information in a manner that is both reliable and accessible. The book's attention to detail, reliance on real-world examples, and commitment to ethical considerations collectively contribute to its overall accuracy as a valuable resource for those engaging in social data analysis.

Relevance/Longevity rating: 4

"Social Data Analysis: A Comprehensive Guide" stands out for its relevance in the field of social research and data analysis. Several key aspects contribute to the book's contemporary and practical relevance:

Integration of Current Data:

The incorporation of data from the 2021 General Social Survey (GSS) ensures that the book's examples and applications are based on recent and relevant datasets. This contemporary approach allows readers to engage with real-world scenarios and analyze data reflective of current social trends. Focus on SPSS and Alternative Tools:

The book's emphasis on using SPSS for quantitative data analysis aligns with the software's widespread use in the social sciences. This focus enhances the book's relevance for readers in academic and professional settings where SPSS is commonly employed. Moreover, the acknowledgment of alternative tools, such as R, adds relevance by catering to a diverse audience with varying software preferences. Practical Applications:

The inclusion of practical examples, screenshots, and step-by-step instructions in the section on quantitative data analysis with SPSS enhances the book's relevance. Readers can directly apply the concepts learned, fostering a hands-on learning experience that is directly applicable to their research or academic pursuits. Ethical Considerations:

The discussion on ethical considerations, particularly the mention of survey weights and their exclusion from the learning dataset, adds relevance by addressing contemporary concerns in research methodology. This ethical awareness aligns with current discussions surrounding responsible and transparent research practices. Diversity of Analytical Approaches:

The book's acknowledgment of alternative methods, such as qualitative and mixed methods data analysis with Dedoose, contributes to its relevance by recognizing the diversity of approaches within the social sciences. This inclusivity allows readers to explore different analytical methods based on their research needs. Supplementary Resources:

The provision of supplementary resources, including the modified GSS Codebook and references to external guides, enhances the book's relevance. These resources offer readers additional tools and information to extend their learning beyond the book, ensuring that they stay updated on best practices and advancements in social data analysis. In summary, "Social Data Analysis: A Comprehensive Guide" remains relevant by incorporating current data, addressing ethical considerations, and catering to a diverse audience with practical examples and alternative tools. The book's contemporary approach aligns with the evolving landscape of social research and data analysis, making it a valuable and relevant resource for students, researchers, and practitioners alike.

Clarity rating: 4

"Social Data Analysis: A Comprehensive Guide" excels in clarity, offering readers a lucid and accessible journey through the intricate landscape of social data analysis. Several factors contribute to the clarity of the book:

Clear Explanations and Language:

The authors employ clear and concise language, making complex concepts in social data analysis accessible to a broad audience. Technical terms are explained in a straightforward manner, enhancing comprehension for readers regardless of their prior knowledge in the field. Step-by-Step Instructions:

The section on quantitative data analysis with SPSS stands out for its clarity due to the inclusion of step-by-step instructions. Readers are guided through processes, ensuring that they can follow and replicate actions easily. This approach fosters a practical understanding of how to apply the theoretical concepts discussed. Visual Aids and Examples:

The use of visual aids, such as screenshots and examples, enhances clarity by providing readers with visual cues to reinforce textual explanations. Real-world examples from the 2021 General Social Survey help readers connect theoretical concepts to practical applications, furthering their understanding. Logical Organization:

The book follows a logical and well-organized structure, moving from introducing social data analysis to specific tools and methods. This logical progression aids in the clarity of the learning journey, allowing readers to build on their understanding progressively. Glossary for Terminology:

The inclusion of a glossary ensures that readers can easily reference and understand key terminology. This contributes to overall clarity by preventing confusion about specialized terms used in the context of social data analysis. Consideration of Different Audiences:

The book is mindful of different audiences by providing options for both students and faculty. This consideration adds clarity by tailoring content to the specific needs and perspectives of these distinct reader groups. Transparency Regarding Limitations:

The book's transparency regarding limitations, such as the exclusion of survey weights from the learning dataset, contributes to clarity. Readers are made aware of the scope and purpose of the dataset, avoiding potential confusion about its applicability to real-world scenarios. In summary, "Social Data Analysis: A Comprehensive Guide" is characterized by its clarity, achieved through clear explanations, practical examples, logical organization, and thoughtful consideration of the diverse needs of its readership. The book effectively demystifies social data analysis, making it an approachable and enlightening resource for individuals at various levels of expertise.

Consistency rating: 4

"Social Data Analysis: A Comprehensive Guide" maintains a high level of consistency throughout its content, ensuring a cohesive and reliable learning experience. The consistency is evident in the uniform and clear language used across chapters, providing a seamless transition for readers as they navigate different sections of the book. The logical organization of topics and the structured approach to quantitative data analysis with SPSS contribute to a consistent learning curve, allowing readers to progressively build on their knowledge. Additionally, the inclusion of real-world examples and visual aids is consistently applied, enhancing the practicality of the book. The authors' commitment to ethical considerations, such as the transparency about the exclusion of survey weights in the learning dataset, reflects a consistent adherence to responsible research practices. Overall, the book's internal coherence, both in language and content, ensures that readers experience a consistent and reliable guide in their exploration of social data analysis.

Modularity rating: 3

"Social Data Analysis: A Comprehensive Guide" excels in modularity, providing a well-organized and modular structure that enhances the learning experience. The book is divided into distinct sections, each focusing on specific aspects of social data analysis. This modular approach allows readers to navigate the content efficiently, catering to different learning preferences and enabling targeted study.

The modularity is evident in the clear demarcation of chapters, from the introduction of social data analysis to the practical application of quantitative data analysis with SPSS and qualitative data analysis with Dedoose. Each section is designed as a standalone module, contributing to a structured and cohesive learning path.

Furthermore, within each module, the book maintains a modular design with sub-sections, ensuring that readers can easily locate and focus on specific topics of interest. The step-by-step instructions provided in the quantitative data analysis section exemplify this modular design, breaking down complex processes into manageable and easily digestible components.

The inclusion of supplementary resources, such as the modified GSS Codebook and glossary, adds to the modularity by offering readers standalone references that complement the main content. This modularity enhances the accessibility of the book, allowing readers to customize their learning experience based on their specific needs and interests.

In conclusion, the modularity of "Social Data Analysis: A Comprehensive Guide" contributes to the book's effectiveness as an educational resource. The well-structured and modular design facilitates a flexible and user-friendly learning experience, making it a valuable tool for readers seeking to navigate the complexities of social data analysis at their own pace.

Organization/Structure/Flow rating: 4

"Social Data Analysis: A Comprehensive Guide" is a well-structured and informative book that serves as an invaluable resource for students and faculty delving into the realm of social data analysis. The authors adeptly navigate readers through the intricacies of both quantitative and qualitative data analysis, placing a specific emphasis on the use of SPSS (Statistical Package for the Social Sciences) for quantitative analysis.

The book begins with a solid foundation, introducing readers to the concept of social data analysis. The initial sections provide a clear understanding of the importance and application of both quantitative and qualitative methods in social research. Notably, the authors strike a balance between theory and practical application, ensuring that readers can grasp the concepts and implement them effectively.

The heart of the book lies in its detailed exploration of quantitative data analysis with SPSS. The authors guide readers through the usage of this powerful statistical software, offering practical insights and step-by-step instructions. The inclusion of screenshots and examples using data from the 2021 General Social Survey enhances the book's accessibility, allowing readers to follow along seamlessly.

Furthermore, the book goes beyond theoretical discussions and provides a modified GSS Codebook for the data used in the text. This resource is invaluable for readers who wish to apply their knowledge to real-world scenarios. The authors' emphasis on the importance of survey weights and their exclusion from the learning dataset demonstrates a commitment to ethical and accurate data analysis practices.

The inclusion of a glossary enriches the learning experience by providing clear definitions of key terms. Additionally, the section on qualitative and mixed methods data analysis with Dedoose broadens the scope of the book, catering to readers interested in a diverse range of analytical approaches.

While the book excels in elucidating complex topics, it does not shy away from acknowledging alternative tools. The authors rightly introduce R as an open-source alternative, recognizing its significance and suggesting that R supplements to the book may be available in the future.

In conclusion, "Social Data Analysis: A Comprehensive Guide" stands out as a comprehensive and accessible resource for individuals venturing into the field of social data analysis. The authors' expertise, coupled with practical examples and supplementary resources, make this book a valuable companion for students, faculty, and anyone keen on mastering the art and science of social data analysis.

Interface rating: 4

The text is free of significant interface issues, including navigation problems, distortion of images/charts, and any other display features that may distract or confuse the reader.

Grammatical Errors rating: 5

The book contains no grammatical errors

Cultural Relevance rating: 5

The text is not culturally insensitive or offensive in any way.

Acknowledgements
How to Use This Book
Section I. Introducting Social Data Analysis
Section II. Quantitative Data Analysis
Section III. Qualitative Data Analysis
Section IV. Quantitative Data Analysis with SPSS
Section V. Qualitative and Mixed Methods Data Analysis with Dedoose
Modified GSS Codebook for the Data Used in this Text
Works Citied
About the Authors

Ancillary Material

About the book.

Social data analysis enables you, as a researcher, to organize the facts you collect during your research. Your data may have come from a questionnaire survey, a set of interviews, or observations. They may be data that have been made available to you from some organization, national or international agency or other researchers. Whatever their source, social data can be daunting to put together in a way that makes sense to you and others.

This book is meant to help you in your initial attempts to analyze data. In doing so it will introduce you to ways that others have found useful in their attempts to organize data. You might think of it as like a recipe book, a resource that you can refer to as you prepare data for your own consumption and that of others. And, like a recipe book that teaches you to prepare simple dishes, you may find this one pretty exciting. Analyzing data in a revealing way is at least as rewarding, we’ve found, as it is to cook up a yummy cashew carrot paté or a steaming corn chowder. We’d like to share our pleasure with you.

About the Contributors

Mikaila Mariel Lemonik Arthur is Professor of Sociology at Rhode Island College, where she has taught a wide variety of courses including Social Research Methods, Social Data Analysis, Senior Seminar in Sociology, Professional Writing for Justice Services, Comparative Law and Justice, Law and Society, Comparative Perspectives on Higher Education, and Race and Justice. She has written a number of books and articles, including both those with a pedagogical focus (including Law and Justice Around the World, published by the University of California Press) and those focusing on her scholarly expertise in higher education (including Student Activism and Curricular Change in Higher Education, published by Routledge). She has expertise and experience in academic program review, translating research findings for policymakers, and disability accessibility in higher education, and has served as a department chair and as Vice President of the RIC/AFT, her faculty union. Outside of work, she enjoys reading speculative fiction, eating delicious vegan food, visiting the ocean, and spending time with amazing humans.

Roger Clark is Professor Emeritus of Sociology at Rhode Island College, where he continues to teach courses in Social Research Methods and Social Data Analysis and to coauthor empirical research articles with undergraduate students. He has coauthored two textbooks, An Invitation to Social Research (with Emily Stier Adler) and Gender Inequality in Our Changing World: A Comparative Approach (with Lori Kenschaft and Desirée Ciambrone). He has been ranked by the USTA in its New England 60- and 65-and-older divisions, shot four holes in one on genuine golf courses, and run multiple half and full marathons. Like the Energizer Bunny, he keeps on going and going, but, given his age, leaves it to your imagination where

Contribute to this Page

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
My Account Login
Explore content
About the journal
Publish with us
Sign up for alerts
Review Article
Open access
Published: 05 April 2024

A systematic review of Stimulated Recall (SR) in educational research from 2012 to 2022

Xuesong Zhai ORCID: orcid.org/0000-0002-4179-7859 1 , 2 na1 ,
Xiaoyan Chu 1 na1 ,
Minjuan Wang 3 , 4 ,
Chin-Chung Tsai 5 ,
Jyh-Chong Liang 5 &
Jonathan Michael Spector 6

Humanities and Social Sciences Communications volume 11 , Article number: 489 ( 2024 ) Cite this article

18 Accesses

1 Altmetric

Metrics details

Science, technology and society

Stimulated Recall (SR) has long been used in educational settings as an approach of retrospection. However, with the fast growing of digital learning and advanced technologies in educational settings over the past decade, the extent to which stimulated recall has been effectively implemented by researchers remains minimal. This systematic review reveals that SR has been primarily employed to probe the patterns of participants’ thinking, to examine the effects of instructional strategies, and to promote metacognitive level. Notably, SR video stimuli have advanced, and the sources of stimuli have become more diverse, including the incorporation of physiological data. Additionally, researchers have applied various strategies, such as flexible intervals and questioning techniques, in SR interviews. Furthermore, this article discusses the relationships between different SR research items, including stimuli and learning contexts. The review and analysis also demonstrate that stimulated recall may be further enhanced by integrating multiple data sources, applying intelligent algorithms, and incorporating conversational agents enabled by generative artificial intelligence such as ChatGPT. This article provides a comprehensive analysis of SR studies in the realm of education and proposes a promising avenue for researchers to proactively apply stimulated recall in investigating educational issues in the digital era.

The science of effective learning with spacing and retrieval practice

Shana K. Carpenter, Steven C. Pan & Andrew C. Butler

A meta-analysis to gauge the impact of pedagogies employed in mixed-ability high school biology classrooms

Malavika E. Santhosh, Jolly Bhadra, … Noora Al-Thani

Preservice teachers’ experiences of observing their teaching competencies via self-recorded videos in a personalized learning environment

Hamdan A. Alamri & Abdulaziz A. Alfayez

Introduction

Stimulated Recall (SR) is an approach commonly used to prompt participants’ retrospection by employing diverse stimuli and interview strategies. This method is frequently applied to examine instructors’ and students’ reflections on their cognitive and affective responses during or after specific educational events or activities (Calderhead, 1981 ; Gass and Mackey, 2016 ). This type of SR represents an effective qualitative method for educational researchers to gather implicit data and has been broadly practiced to investigate various teaching and learning occurrences, including teacher cognition, study strategies, and language learning (Meade and McMeniman, 1992 ; Van der Kleij et al., 2017 ; White et al., 2016 ; Sundberg et al., 2018 ; Martinelle, 2018 ; Cao et al., 2019 ; Martinelle, 2020 ). Moreover, in addition to serving as a research tool to explore instructors’ and learners’ internal thoughts, several studies have innovatively implemented SR as a teaching and learning strategy to foster students’ metacognition (Zhai et al., 2018 ; Jensen, 2019 ). Nevertheless, although the purposes of SR-enabled research appear to be diverse, there are reasons to extend its use in more educational and research settings.

The vast technology integration into education has ushered in changes in the selection of stimuli and technologies for adopting SR in educational research (Gazdag et al., 2019 ). Technological advancement applied in teaching and learning settings have also expanded the sources of stimuli beyond traditional written notes, classroom photographs, and video recordings. Participants’ learning records on digital platforms and mobile devices can also be used as stimuli to evoke the memory of their own learning path (Koltovskaia, 2020 ; Lindfors et al., 2020 ). Furthermore, the shift of instructional environments from offline to online has rendered educational activities in physical scenes more static, lacking observable interactivity to generate an effective stimulus (Duo and Song, 2012 ; Gijselaers et al., 2016 ; Tan et al., 2021 ). Some studies have leveraged physiological feedback signals such as eye movement, setting position, and EEG data to provide valuable cues about changes in learners’ inner thoughts (Zhai et al., 2018 ; El and Windeatt, 2019 ). However, owing to the constantly evolving technological landscape in learning environments and pedagogical strategies, the question of whether traditional stimuli need to be improved and how to choose new stimuli remains unresolved (Wijayasundara, 2020 ).

When practicing interview strategies, researchers have exhibited distinctive tendencies in time arrangement and questioning techniques (Gass and Mackey, 2016 ). Even when the same stimuli were selected, the adoption of interview strategies varied across studies. Concerning the time arrangement of the interview, most researchers contend that participants should be presented with the stimuli and interviewed immediately after the instructional activity, while some researchers intentionally introduce an interval before further interviewing (Gass and Mackey, 2000 ; Kurki et al., 2016 ). In terms of questioning techniques, interviewers’ questions can be either entirely open-ended or focused, depending on the research design and educational settings. For instance, Heikonen et al., ( 2017 ) commenced with general questions and subsequently narrowed the question scope to explore student and instructors’ reflections on classroom incidents. In contrast, Hu and Gao ( 2020 ) posed rather specific questions on students’ responses to linguistic challenges in learning science through English. These disparities may be attributed to the distinct subjects and research questions that SR measures aim to address (Jackson and Cho, 2018 ; Tiainen et al., 2018 ).

In light of the ongoing developments in education and technology, it is worthwhile to conduct a meticulous review of the latest research on applying SR methods in education. Previous reviews were either outdated or narrow in scope. For instance, Keith’s ( 1988 ) review centered on studies that applied SR to investigate instructors’ cognitive processes, which, although valuable at the time, can only provide limited guide for current applications of SR in education. More recently, Gazdag et al., ( 2019 ) reviewed 35 articles on the use of Video Stimulated Recall (VSR) to enhance instructors’ reflective thinking. However, this study’s scope was confined to implementing VSR in teacher training and excluded studies in broader educational settings. Therefore, further studies are needed to comprehensively examine the application of SR across diverse contexts.

The present study offers a comprehensive review of research using SR in manifold teaching and learning contexts over the past decade. The investigation scrutinizes the characteristics of these studies, such as their research aims, stimuli, and interview strategies. It examines the interplay among these elements, including variations in the purposes of SR employment across disciplines. The ultimate goal of our study is to provide valuable insights for future applications of SR in education and also to aid researchers in exploring the external behaviors and internal thought processes of both instructors and students in a more effective manner.

Literature review

The theoretical foundation of sr in education.

SR is a research technique inspired by Dewey’s ( 1933 ) reflective thinking concept, which involves presenting participants with vivid prompts to evoke their memories of an original scenario (Bloom, 1953 ). Since its inception by researchers at Stanford University in 1970, SR has been an essential tool in pedagogical research and widely adopted to investigate various teaching and learning activities in educational research (Stough, 2001 ). Typically, SR comprises two stages: presenting stimuli and proposing recall questions (see Fig. 1 ) (Chu and Zhai, 2023 ). Researchers select specific artifacts, such as notes, audio or video recordings, that exhibit participants’ behavior or cognitive tasks as stimuli, followed by interviews that prompt participants to articulate their intrinsic thoughts, mental processes, or individual feelings at the moment when the stimuli were generated (Calderhead, 1981 ; Lyle, 2003 ).

The figure shows the main stages of presenting stimuli and proposing recall questions when applying SR.

The theoretical basis of SR in educational research draws on the Retrocue Effect and the Cognitive Theory of Multimedia Learning (CTML) (Mayer and Moreno, 1998 ; Moreno and Mayer, 1999 ; Souza and Oberauer, 2016 ; Shepherdson et al., 2018 ). The Retrocue Effect, a cognitive psychology theory, suggests that an individual’s visual working memory is enhanced when their attention is directed toward prior information, even after a delay or distraction (Souza and Oberauer, 2016 ). Neuroscientific and biopsychological research both provide evidence supporting the protective effect of retroactive attentional focusing on working memory (Duarte et al., 2013 ; Schneider et al., 2017 ). According to this theory, retro cues, such as visual stimuli, improve the quality of retrieval and cognitive processes while also reducing cognitive load effects (Shepherdson et al., 2018 ). Based on this mechanism, SR can offer accurate and specific insights into an instructor or a learner’s thoughts and attitudes towards educational tasks.

In addition, the Cognitive Theory of Multimedia Learning (CTML) suggests that multimedia learning is most effective when information is presented in both visual and auditory formats, as learners are actively engaged in the learning process (Mayer and Moreno, 1998 ; Moreno and Mayer, 1999 ). As described in the CTML, learners have two separate channels for processing information: visual and verbal (Mayer, 2002 ; Mayer and Moreno, 2003 ). When multiple forms of stimuli are presented during the SR interview, instructors and learners become more cognizant of their prior experiences in each channel, which helps them articulate their thought processes in greater detail and enhances their retrospection of previous knowledge and cognition. In conclusion, the application of SR in educational research is rooted in the principles of the ICT and the CTML. Implementing SR provides researchers and practitioners with a valuable tool to gain insight into learners’ and instructors’ cognitive processes, ultimately leading to more effective teaching and learning.

The educational application using SR

The SR method is an effective technique used in qualitative educational research to gather data on instructors’ and learners’ thought patterns related to specific events. This method allows researchers to explore instructors’ and learners’ thinking and decision-making processes, making it a valuable tool for data collection (Nguyen et al., 2013 ; Bowles, 2018 ). The use of SR in educational research is critical for maintaining internal validity, as it provides introspective data. Additionally, SR has broad applicability and can be employed in various disciplines for a range of research aims (Meade and McMeniman, 1992 ; Kurki et al., 2016 ; Yu and Hu, 2017 ; Rietdijk et al., 2018 ; Martinelle, 2020 ). For instance, Yu and Hu ( 2017 ) used SR to probe second language learners’ intrinsic and personalized perceptions of peer feedback in collaborative writing assessment, by exploring students’ learning behaviors through interviews. Similarly, Kurki et al. ( 2016 ) and Rietdijk et al. ( 2018 ) tapped into SR to explore how instructors use various teaching strategies and their underlying beliefs, particularly concerning non-cognitive dimensions such as social and emotional factors.

Aside from its application as an educational research method, SR can also serve as an effective teaching and learning strategy. Instructors can use SR to assess what learners remember or may have overlooked to determine learning reinforcement strategies. SR enables learners to examine and articulate their thoughts through memory retrieval and thus elevating their thinking to a new level of expression. Therefore, SR can enhance learning rather than solely serving as a research approach (Smagorinsky, 1998 ; Egi, 2008 ). In addition, VSR is a valuable teacher training and development tool that includes video-supported reflection and questioning. This approach motivates instructors to reflect on themselves and their practice consciously, facilitates metacognitive reflection among preservice teachers, and provides reflective prompts for educational interactions (Geiger, Muir, and Lamb, 2016 ; Endacott, 2016 ).

While linguistics and teacher education are the primary application areas of SR, it is also used in other subjects, such as STEM education (Gass and Mackey, 2016 ; Al Mamun, Lawrie, and Wright, 2020 ; Schindler and Lilienthal, 2019 ). However, the purpose of applying SR varies depending on the subject and learning environment. Recent advances in instructional technologies have transformed teaching strategies and educational settings, yet the relationship between these elements and the principles of SR application in distinct contexts is still ambiguous.

S timuli and interview strategies in SR

The rapid diffusion of Information and Communication Technology (ICT) and the exponential growth of online learning have brought new challenges and opportunities for using SR in educational research and in teaching and learning. Integrating ICT into education requires a careful selection of stimuli that can adapt to the constantly evolving learning environments. When applied in physical environments, audio or video stimuli respond favorably to interactions between teachers and students, enabling subsequent interviews to investigate their inner impressions or perceptions (Chu and Zhai, 2023 ). In contrast, educational activities incorporating digital technologies are not easily observable, with instructional behaviors conducted through electronic devices and in video or audio conference systems. It is often challenging to find informative stimuli reflecting teacher-student interactions in digital settings.

Nevertheless, technological breakthroughs have enriched stimuli by expanding data collection channels and capacities at the same time. Through the integration of additional stimuli sources such as weblogs, computer screen captures, and biofeedback data, researchers are able to unearth information about learners’ inner workings. For example, Révész et al., ( 2017 ) gained a comprehensive picture of the L2 writing process and acquired a deeper understanding of implicit thinking using eye movement data-based stimuli. Overall, considering the diverse data collection methods and changing learning contexts, stimuli selection in SR in technology-enabled schooling still requires further clarification.

The interview stage is another crucial aspect of SR that distinguishes it from conventional memory recall. This stage emphasizes estimating internal thinking processes and determining how the method can encourage instructors’ and learners’ reflection and delve into their internal ideas. During the interview stage, researchers mainly acquire tacit data. Previous studies are inclined to perform interviews promptly after class and employ standard open-ended questions to encourage participants’ agency in reflecting on their experiences (Gass and Mackey, 2000 ; Chu and Zhai, 2023 ). However, some studies have chosen different approaches. For example, when investigating early childhood teachers’ instructional activities, behavior, and emotions, Kurki and his colleagues (2016) delayed inviting teachers to take part in the interview by two weeks. Additionally, researchers argue that, apart from using generic questions, incorporating specific follow-up questions that closely align with the research aim is equally crucial (Heikonen et al., 2017 ; Hu and Wu, 2020 ). Despite the significance and disparities in interview strategies, few studies have specifically analyzed this issue, and well-developed principles of organizing interviews and questions is absent.

SR has become an essential technique for examining cognition and behavioral patterns in education by activating instructors’ and students’ retrospection through stimuli and interviews. As SR has evolved and educational paradigms have transformed, the research purpose and critical steps, such as stimuli selection and interview strategies, of applying SR in educational research require further discussion. Education is a complex system with intertwined intrinsic elements such as discipline differences and learning environments (Jacobson and Wilensky, 2006 ; Jacobson et al., 2019 ), which can influence stimuli preference and the conduct of interviews.

Therefore, in order to provide insights for learning from past educational applications of SR and enhancing future developments, the present systematic review scrutinized the evolution of the SR method in educational research from 2012 to 2022. It aimed to elucidate what the contributions SR has made, how SR has been implemented, and the challenges and potentials it presents. To fulfill these objectives, we further proposed six specific coding items (see Table 1 ) to guide our content analysis coding procedure and decoding interpretation.

Guided by the aforementioned research questions, we systematically analyzed and interpreted studies related to SR from 2012 to 2022. Given the significant changes in teaching environments and research methods associated with the rapid development in educational technology, we believe that a 10-year time span can provide sufficient coverage of research in a variety of disciplines. We used qualitative content analysis to examine these studies, which consists of two steps: selecting papers for review and coding these papers by using an established coding scheme.

Paper selection

To guarantee the quality of selected, our research team reviewed well-recognized peer-reviewed articles in the Web of Science (WOS) core collection, Scopus, and IEEE Xplore. These databases contain reputable journals with recognized impact factors. The articles retrieved in WOS and Scopus can be further refined into social science or educational categories, allowing for more precise retrieval. Additionally, given the focus of this research on the use of technology in education, the IEEE database provides focused research in scientific and technical disciplines.

Two stages comprise the processes used to identify the research papers. In the first stage, the keyword “SR” was selected, and the discipline was refined to “education and educational research”. This process yielded 309 articles. In the second stage, the abstracts and full text of the chosen articles were manually and systematically screened by researchers to confirmed that they: (1) included the SR protocols, (2) prompted participants to reflect on their thinking process, (3) focused on research issues in the field of education, and (4) provided empirical evidence or evaluation rather than solely summarizing previous findings. For example, some articles merely reviewed others’ research on the SR method employed in teaching settings or using painting-based stimuli to spark students’ prior knowledge did not meet the inclusion criteria (Gazdag et al., 2019 ; Walan and Enochsson, 2019 ).

According to Golhasany and Harvey’s ( 2023 ) study, the coder should pose doctoral degrees or professorships in the relevant field, and each identified papers should be individually scrutinized by experts. Finally, three experts were selected to examine the sample pool: two of whom have doctoral degrees and professorships on learning technology, while the third have a doctoral degree in educational management and post-doctoral experience on learning technology. Moreover, to ensure there are no conflicts of interest, only one coder is involved in the authorship. The inter-coder reliability was assessed following a specific schedule: first, the coders independently examine the selected samples and provided their judgment. Then we use the Fleiss Kappa test in SPSS 26 to test the reliability. The results ranged from 0.874 to 0.973 indicating satisfactory inter-rater reliability and consistent coding for each item. Finally, we adopted the final coding results if all the experts or at least two of them agreed. Finally, the research team identified 257 representative papers as the research sample of this study. As recommended by Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA, Page et al. 2021 ), we conducted the systematic review with a strong and thorough methodology. Figure 2 depicts the flow of this screening process, which is in accordance with Moher et al. ( 2009 ).

The diagram presents the systematic review flow according to PRISMA.

Coding procedure

The identified articles were systematically coded to carry out a precise and thorough examination of the utilization of SR in education. By adopting Gass and Mackey’s ( 2000 ) definition of SR, this study identifies its key components. They established a coding framework, including the research aim, stimuli, questioning technique, and questioning interval. Additionally, coding the learning subject and educational context helped clarify how to implement SR effectively in various situations. Table 2 illustrates the background information of SR research, such as the learning subject and educational context. Despite reviewing research involving instructors and students as subjects, this study did not differentiate between these two groups as the primary focus of SR is to investigate the participants’ consciousness and thinking behind their behavior, regardless of their roles. The table included in supplementary information described our data collection process.

The coding process involved identifying and extracting relevant data from the selected papers. Any discrepancies were resolved through discussion and consensus-reaching among the research team. We then analyzed the coded data and identified patterns and themes abiding by the content analysis method. The findings of this review are presented in the following sections, addressing the research questions outlined earlier.

Results and discussion

In accordance with the content analysis and coding criteria mentioned above, 257 papers were thoroughly examined. The following sections present the results and provide a corresponding discussion of the research questions.

RQ1: Research aims

The current literature reveal that SR is often applied to studying instructors’ and learners’ inner thoughts and ideas, prompting them to recall and comment on their thinking process. Furthermore, this approach can also examine the effect of teaching and learning strategies and to improve participants’ metacognitive skills. Because SR has long been used in educational settings, it is surprising that more substantial research has yet to be conducted on how it might be expanded and how to overcome its limitations such as time factors and distractions. Therefore, our work focuses on promoting the effectiveness of widespread application of SR in teaching and learning.

Exploring thought patterns

Exploring thought patterns is the primary focus of most educational research that uses SR. This includes investigating non-cognitive and cognitive processes, as well as higher-order thinking. As shown in Table 3 , over half (157 in total) of the research reviewed employed SR to achieve this objective. Eighty-two of the reviewed studies explored patterns of non-cognitive processing, such as motivation, emotions, and cultural orientation (Lichtinger and Kaplan, 2015 ; Ucan and Webb, 2015 ; Wilby et al., 2017 ). This method can also investigate the variables influencing the willingness to communicate or the ethical considerations of instructional practices in language learning (Rissanen et al., 2018 ; Chichon, 2019; Peng, 2020 ).

In addition, fifty-five studies explored patterns of cognitive processing, probing the epistemic thinking of diverse participants in various subjects, including language learning, STEM, and the arts (Bogard et al., 2013 ; dos Santos and Loveridge, 2014 ; Révész et al., 2017 ). Furthermore, some studies based on SR also obtained insight into both cognitive as well as non-cognitive processes through the integration of multimodal data (Lambert and Zhang, 2019 ). It is worth mentioning that a total of 11 papers explored both cognitive and non-cognitive thought processes with SR.

Finally, nine studies applied SR to investigate patterns of higher-order thinking, such as creative thinking and critical thinking, as well as the collaborative process (Rissanen et al., 2019 ; Schindler and Lilienthal, 2020 ; Łucznik and May, 2021 ). The application of SR in these studies allowed for a more comprehensive understanding of participants’ thinking processes and the factors that contribute to effective collaboration and higher-order thinking.

Investigating the effect of educational strategies

Another purpose for research employing SR is to investigate how participants’ learning processes and experiences are affected by instructional design or learning models. Specifically, 35 articles used SR to investigate the impact of specific learning strategies in educational settings, yet 54 studies examined instructional techniques. It seems that SR can facilitate investigating how instructors and students understand and apply newly adopted teaching or learning strategies.

For instance, SR has revealed the pedagogical knowledge base related to the use of dashboards and the provision of feedback by novice teachers (Karimi and Norouzi, 2017 ; Molenaar and Knoop-van Campen, 2018 ; Yu, 2021 ). In terms of the effectiveness of learning techniques, such as computer-enhanced self-directed learning, SR offers a more precise and comprehensive understanding from students’ viewpoints (White et al., 2016 ; Deng, 2020 ; Chu and Zhai, 2023 ).

Extensive empirical studies have shown that data acquired through SR can enhance the interpretability of single-outcome data, such as test results, and also produce more insightful information to evaluate and enhance strategies for improved learning outcomes for both instructors and students (Van der Kleij et al., 2017 ). In these studies, SR provided a deeper comprehension of how instructional design or learning strategies impact participants’ learning experiences and outcomes.

Improving metacognition

Nine articles took advantage of SR to improve participants’ metacognition. One example is using SR in online language learning, where learners can compare feedforward and eye-movement data to develop their metacognitive skills (Zhai et al., 2022 ). Metacognition refers to an individual’s awareness of their thinking processes and understanding of the underlying patterns (Flavell, 1979 ). Educational psychologists have widely acknowledged the significance of metacognition due to its substantial correlation with learners’ academic achievements.

Metacognitive activities usually occur during the self-reflection phase and involve the participants’ evaluation of their own cognition, understanding, and task performance. Using recorded learning processes as stimuli, participants are prompted to explain or evaluate their past behavior instead of simply recalling knowledge. Encouraging student participation in video-stimulated recall conversations enhances their self-reflection and improves their metacognitive skills by giving them a scaffold (Van der Kleij et al., 2017 ).

RQ2: Stimuli

Regarding the stimuli used to arouse recall, video recordings of the learning process have gained tremendous popularity. While some changes have occurred during the evolution of SR methods, such as the improvement of video stimuli and the integration of physiological feedback data.

Optimizing video stimuli

Video recordings are a widely popular type of stimuli in educational SR research, as evidenced by nearly 70% of the reviewed studies (173 articles) that utilized them. This prevalence of video stimuli has been noted in previous review research by Gazdag et al. ( 2019 ), which to some extent, explains the exclusive focus on video stimuli in his study. These recordings commonly consist of real-life scenes from classrooms, laboratories, and screen captures of technology-mediated learning settings. To serve as effective incentives for participants, video recordings should reflect the interactions between instructors and learners, and researchers ought to regulate their length to prevent participant weariness (Lee, 2020 ).

A number of enhancement options have been suggested by researchers. One technique is to use multiple cameras to record and display split-screen videos, providing various perspectives of the learning environment. For instance, Jackson and Cho ( 2018 ) produced a split-screen video recording of teachers’ and students’ simultaneous behaviors, enabling a more potent stimulus for supporting event recall, contextual and situational recall.

Additionally, some researchers have used head-mounted video cameras to record video from the participant’s perspective, visually reproducing original learning scenarios. Such an approach is beneficial in studies examining interpersonal communication, such as those exploring teacher-student interactions or teacher interventions in early childhood peer conflict (Agricola et al., 2021 ; Myrtil et al., 2021 ).

Utilizing biofeedback data

With more accurate and detailed data, biofeedback data (14 articles) is increasingly considered a stimulus choice for self-reflection. Currently, eye-tracking technology is the most commonly used physiological feedback technique. The Eye-Mind hypothesis suggests that eye movements correspond to mental operations, allowing researchers to infer cognitive processes from gaze patterns (Obersteiner and Tumpek, 2016 , p. 257). By combining eye-tracking data with self-reflection, potential ambiguity and uncertainty in eye-tracking techniques are reduced, giving a more thorough overview of the educational process for reflection (Schindler and Lilienthal, 2019 ; El and Windeatt, 2019 ; Chu and Zhai, 2023 ).

Moreover, other physiological indicators have served as informative stimuli in self-reflection. Zhai et al. ( 2018 ) found that online learners’ reading comprehension and cognitive abilities were significantly improve by using both eye-movement and EEG physiological signals as stimuli. Multiple physiological indicators can be included to provide a more thorough and accurate picture of the cognitive and affective states of learners during the learning process.

RQ3: Time factors

Considering time factors is crucial for the utilizing SR method in educational research. This is because time not only influences the selection and processing of stimuli but also has implications for the subsequent interviews. Specifically, enhancing the temporal properties involves both reducing the presentation time and increasing the span of information provided by the stimuli source. Moreover, it is essential to set appropriate time intervals to schedule the interviews. The reviewed literature suggests that the interview schedule may vary depending on the study.

Enhancing the temporal properties of stimuli

Presenting learners with stimuli is intended to assist them in reflecting on their previous learning activities. Nevertheless, if the presentation of stimulus sources persists for too long, it can also impose a heavier cognitive load on learners (Pratt and Martin, 2017 ). In general, stimuli sources in textual, image, and other static formats are more convenient due to their controllable presentation duration for participants. However, for classroom video recordings stimuli, direct video replay may take a considerable amount of time. Considering the time spent, one such approach involves selecting clips from full-length video footage, reducing the duration of the stimuli, and enabling participants to concentrate specifically on behaviors that are pertinent to the research aim (Määttä et al., 2016 ).

The time span of stimuli is also not limited to the classroom. As demonstrated in the 16 studies reviewed, combining multiple data sources has proved more effective. The integration of various materials, including text, video, and pictures, enhances the information capacity and authenticity of the recorded details. For instance, in limited interaction scenarios, researchers often use think-aloud methods, allowing participants to verbalize their thoughts, along with the videotapes, to augment the information provided (Kang and Pyun, 2013 ).

In addition, incorporating stimuli from multiple sources can encompass both subjective and objective aspects. Video recordings only capture a limited timeframe, while learning behaviors extend beyond the confines of the classroom. Cues to stimulate participants’ recall can also come from guide sheets, teacher preparation notes, and student class notes (Chu and Zhai, 2023 ). In an investigation on the use of metacognitive interventions in twenty-first century writing pedagogies, stimuli included a classroom tour video, a literacy autobiography, a teaching plan, and other instructional materials (Jensen, 2019 ).

Setting up flexible intervals

The time interval between in-class instructional activities and SR interviews generally varies across researchers. Among the 149 reviewed studies where the time interval was specified, the majority of the study (126) preferred instant reflection. Instant reflection involves conducting SR interviews as soon as participants finish their learning tasks, typically with only a 5- or 10-min interval or a slight delay according to the timetable for curriculum (White et al., 2016 ; Rassaei, 2015 ; Shintani, 2016 ; Fernandez, 2018 ).

A shorter time span makes sure that participants recollect the task’s cognitive processes more precisely, which improves the accuracy of the interview results (Gass and Mackey, 2000 ). Instant reflection is particularly valuable in studies that require precise information about the learners’ cognitive processes and strategies during the learning task.

However, some researchers (23 studies) purposefully extended the time interval between instruction and recall, for example, 2–4 weeks after the task was completed (Harvey et al., 2014 ). This design may optimize the study by allowing more time for the process of previously recorded raw data and footage (Nurmukhamedov and Kim, 2010 ; Kurki et al., 2016 ). Delayed interviews can also reduce research impact on participants by avoiding interference with subsequent teaching and learning activities (Dos Santos and Hentschke, 2011 ).

RQ4: Interview strategies

During the interview phase of SR, to better guide participants in autonomously reflecting on the teaching and learning process, researchers also need to pay attention to the use of strategies, including the openness and value-oriented nature of the questions.

Posing appropriate questions

SR interviews are utilized to encourage reflective thinking in participants within an open and dialogic environment through questioning strategies. Typically, this kind of interview consists of a succession of open-ended, introspective, and generic questions that do not require predetermined answers. This pattern has been observed in 88 reviewed studies, including research merely posing general questions, as well as those starting with general questions and then progressively narrows down the focus. During these interviews, researchers should take on the role of listeners, serving to train, facilitate, and illuminate while avoiding asking leading questions that could result in biased responses (Ramnarain and Modiba, 2013 ; Egi, 2008 ; Gass and Mackey, 2000 ; Sato, 2019 ). For instance, researchers should avoid yes-no questions that could encourage participants to react a specific way or provide presentational responses. This approach ensures that the purpose of the SR interview is maintained and that the risk of biased responses is minimized (Thararuedee and Wette, 2020 ; Rassaei, 2020 ;).

While questioning in SR interview should leave ample room for participants to retrospect, it must also address the research questions. Thus, 25 papers suggest that questions should be open-ended at the beginning but become increasingly specific as the interview progresses (Stolpe and Björklund, 2013 ). Researchers can use supplementary queries as prompts to ensure that the interview stays on topic and delves deeper into the research questions, depending on participants’ responses (Qiu and Lo, 2017 ; Qiu, 2020 ; Chu and Zhai, 2023 ).

Staying value-neutral in guiding

In addition to the scope of questioning, the neutrality and guidance provided by the interviewer are crucial. Participants receive training before the interview on how to reflect on previous cognitive processes, and the interviewer should remain as neutral as possible during the interview to capture retrospective thinking solely supported by the stimuli (Consuegra et al., 2016 ). If respondents feel that the questions are biased or contain value judgments, they may feel pressured to rationalize or make up explanations, leading to inaccurate reporting of their thoughts. Therefore, the interviewer must carefully design questions wording and adopt a supportive attitude that indicates curiosity in the descriptions provided by participants rather than making judgments (Wu, 2019 ; Schindler and Lilienthal, 2019 ).

RQ5: The relationship between different coding items

In addition to key application procedures such as research aims, stimuli, time factors, and interview strategies, the implementation of SR in educational research is also influenced by intrinsic factors within the educational context, such as learning subjects and educational context. The results of the review (see Table 3 ) indicate that SR is primarily employed within the fields of linguistics (115 articles) and teacher education (48 articles), with relatively fewer studies in areas such as the arts (9 articles). Over 75% of the articles still apply SR in physics learning environments, while nearly 20% explore the use of SR in online platforms.

To enhance the exploratory nature of the research discussion, the current study delved deeper into the intricate relationship between coding items. It is important to note that only outcomes warranting further exploration and discussion are presented in the subsequent section.

The relationship between research aims and learning subject

This bubble chart (Fig. 3 ) illustrates the connection between research aims and learning subjects, with the size of the bubbles indicating the number of relevant papers reviewed. Our current analysis aimed to explore whether SR is more suitable for investigating specific research questions in different disciplines.

The relationship between research aims and learning subjects is depicted in this bubble chart, where the size of the bubbles represents the quantity of relevant studies.

Regarding research aims, SR was primarily used to investigate patterns of non-cognitive processing and the effect of instructional strategies across all subjects. In linguistics, researchers most frequently utilized SR to explore patterns of cognitive processing (29 articles), non-cognitive processing (29 articles), and learning strategies (26 articles). Another six studies focused on both cognitive and non-cognitive occurrences in linguistic teaching and learning. These findings are consistent with prior research highlighting the importance of non-cognitive factors (e.g., motivation) and learning strategies in language learning (Lin et al., 2017 ). Furthermore, 20 studies using SR investigated non-cognitive elements in teacher education contexts where teachers’ non-cognitive factors, such as intrinsic motivation, are strongly associated with their professional development (Maaranen et al., 2019 ).

In the realm of educational subjects, SR has also occupied a pivotal within the domain of STEM and art education research. Within the STEM disciplines, researchers have employed this methodology to probe the impact of pedagogical strategies (13 articles), non-cognitive processing (10 articles), and cognitive processing (6 articles). Intriguingly, SR has been invoked more frequently to investigate cognitive rather than non-cognitive factors within the sphere of art education (four articles versus three). This inclination may stem from the intricate nature that cognitive processing exhibits in artistic creation (Révész et al., 2017 ). Nonetheless, SR has demonstrated its utility as an effective facilitator, enabling arts educators to acquire profound insights into the cognitive aspects of art instruction and learning. For instance, dos Santos ( 2018 ) documented music teachers’ approaches to the instruction of rhythmic skills as stimuli, facilitating their reflection upon their cognitive processes and their utilization of their didactic content knowledge.

Linguistics and teacher education are two fields that more frequently took advantage of SR as a teaching strategy beyond research methods (Meade and McMeniman, 1992 ; Geiger et al., 2016 ; Sanchez and Grimshaw, 2019 ). Four articles in linguistics and three in teacher education explore using SR to improve participants’ metacognition. In particular, teacher’s professional development and language learning emphasize reflective practice and metacognition (Belvis et al., 2013 ; Zahid and Khanam, 2019 ). For example, research on teachers’ noticing highlights the importance of their cognition and behavior in classroom situations (Chan et al., 2021 ; Amador et al., 2021 ). In language learning, metacognitive awareness has been found to enhance foreign language writing ability, emphasizing the need for metacognitive strategies to improve learners’ skills (Ramadhanti and Yanda, 2021 ; Farahian, 2015 ). These requirements for introspective behavior and metacognition in language learning and teachers’ professional development align with the essential steps and reflective characteristics of SR.

The relationship between stimuli and educational context

The bubble diagram (Fig. 4 ) displays the stimuli and learning environment, with the size of the bubbles corresponding to the number of articles in the review. Our analysis aimed to scrutinize which stimuli are commonly adopted in different learning environments.

With the size of the bubbles indicating the number of articles in the review, the bubble diagram illustrates connection between learning environment and stimuli.

Firstly, video footage remains the dominant stimulus across various scenarios, with over half of the studies (141 articles) utilizing video in physical learning settings and 23 studies using video footage to stimulate reflection in digital learning platforms and OMO settings. However, there is a disparity in the type of videos used in these settings. Physical learning environments mostly relied on live-action videos that authentically recorded participants’ behaviors and interactions (Chan and Yung, 2015 ; Nyberg and Larsson, 2017 ), whereas digital learning environments utilized device screen recordings that captured participants’ operations on computer-supported learning platforms (Rassaei, 2013 ; Lee, 2020 ).

Secondly, alongside video recordings, physiological data of participants is most frequently used as a stimulus for SR research based on online platforms (10 articles). This trend is reasonable as screen recordings alone may not fully reflect students’ behavior, mainly if they do not perform mouse manipulation or keyboard input. For instance, eye-gaze behaviors provide direct and objective evidence, including fixation duration, fixation count, and scanning path, allowing for stronger conclusions about learners’ cognitive processes and learning strategies (Lai et al., 2013 ; Michel et al., 2020 ).

Thirdly, the diagram indicates that studies utilizing SR in physical environments are more mature and inclined to utilize multimodal stimuli. However, studies in online platforms, OMO environments, and VR environments are still limited, with predominantly homogeneous stimuli. Only one study explored students’ learning strategies in an English task using video footage as stimuli in a virtual reality setting (Park, 2018 ). Thus, more than relying on text, images, or audio and video alone as stimuli is required, and more physiological and multimodal stimuli should be employed in future teaching and learning environments.

The relationship between questioning strategy and research aim

The diagram presented here (Fig. 5 ) displays the questioning strategy and purpose of the study using the SR method, with the size of the bubbles representing the number of articles. Based on this information, our analysis aimed to explore whether the purpose of the study influenced questioning strategies.

With the size of the bubbles signifying the number of articles, the diagram illustrates the SR questioning strategy and the research aim.

They were excluding articles that did not mention questioning techniques, 23 studies exploring non-cognitive processing utilized general questions during SR, outnumbering studies that implemented more focused questioning strategies (20 articles). This preference for general questioning may stem from the diverse and individualized nature of non-cognitive skills, which include motivation, responsibility, and perseverance (Smithers et al., 2018 ). Consequently, general questions are more appropriate as they allow participants to autonomously recall non-cognitive processing activities with the aid of stimulus materials. Moreover, reflection on non-cognitive processing is prone to interference from external factors. If interview questions are too directed towards the research objectives, they may interfere with the results.

In contrast, studies focusing on cognitive processing patterns predominantly utilized questioning sessions centered on research questions (21 articles), nearly double the number of studies using general questioning strategies (14 articles). Cognitive processing is often intimately related to the teaching or learning activity. Thus, researchers tend to focus their questioning on the learning activity that concerns the research goals. Notably, one article exploring cognitive patterns adopted a different questioning technique: focused first and then general. This article investigated what musicians learned when teaching older adults (Perkins et al., 2015 ). On the one hand, the research questions themselves were exploratory, and the researcher expected participants to provide more cognitive information. On the other hand, this phenomenon may also reflect the divergent and creative thinking of art learning, requiring questions that encourage participants to reflect freely after satisfying the research objectives.

RQ6: Potential improvements

In addition to exploring how effectively employ SR in education, the current review also points to possible future directions on SR research with existing models of computer-supported learning and technology-assisted instruction.

Enhancing the dependability of outcomes through the synthesis of multifaceted data sources

Table 3 shows that a mere fraction under 10% of the studies (16 articles) utilized multi-source stimuli. Indeed, the amalgamation of data derived from disparate stimuli can provide a complementary and robust scaffold for the outcomes of SR. This is attributable to the fact that the integration of heterogeneous types of stimuli broadens the information spectrum, providing participants with supplementary prompts that facilitate the recollection of cognitive processes. Such an approach diminishes the cognitive load on subjects, assists them in articulating more accurate reflections, and arguments the reliability of SR. Furthermore, this practice contributes to the transparency of educational research (D’Oca and Hrynaszkiewicz, 2015 ). For instance, combining video, audio, and text stimuli can offer a more comprehensive and nuanced understanding of learners’ cognitive processes and behaviors.

Additionally, using multimodal stimuli can help address the limitations of using a single type of stimuli and enhance the ecological validity of the study by better replicating real-world learning environments. Some researchers (e.g., Rankanen et al., 2022 ) conducted a study that employed both quantitative and qualitative methods to investigate the impact of non-instructional clay-making in art education on learners’ creative thinking and positive emotion stimulation. By combining multiple data sources, including physiological data such as heart rate variability (HRV) and electrocardiogram signals, this study provides a more detailed understanding of the art experience and the mechanisms at work in different art forms. Unlike previous research that relied solely on questionnaires, this study includes more objective and in-depth quantitative data analyses of art learning tasks. Additionally, the researchers used video-stimulated recall in addition to HRV data to provide a comprehensive perspective on the learners’ experience of non-instructional clay-making in art education. Including qualitative data can reveal the positive or negative value of the emotional experience of artmakers and provide a more nuanced understanding of the emotional complexity of art.

Strengthening the acquiring and processing of stimuli by adopting intelligent algorithms

The synthesis of the review indicates that over 70% of the studies (185 articles) employ video recordings as a singular or combined source of stimuli as depicted in Table 3 . Therefore, the employment of AI algorithms to refine the processing of video stimuli could markedly enhance the application of SR in education research.

Firstly, algorithm-supported techniques can assist in selecting the relevant learner interaction portion of video stimuli, thus shortening the length of SR and automatically extracting key information. Wass and Moskal ( 2017 ) proposed an automatic video annotation tool, which scaffolds more profound reflections and reduces the cognitive load in participating instructors and students. This intelligent excerpting and annotation process saves time and reduces labor, thus improving the efficiency of SR. Furthermore, algorithm-supported techniques can help to automate the coding and analysis of the data, reducing the potential for human error and increasing the reliability of the findings.

Intelligent algorithms can effectively address the challenge of identifying specific moments or events in classroom videos that are relevant to research questions and require meticulous observation, particularly in cases where the video playback duration is extended. Recent advances in computer vision and machine learning have made it possible to automatically extract valuable information from classroom videos, such as the head pose, gaze direction, and facial expressions of instructors and learners, as well as synchronous behaviors between neighboring students (Goldberg et al., 2021 ). Furthermore, the use of Convolutional Neural Networks (CNN) and Deep Neural Network (DNN) enables the analysis of audiovisual data to identify and annotate class environments, such as the teacher’s instructional strategies, student engagement, and classroom management (Ramakrishnan et al., 2023 ). The application of these intelligent algorithms has significant implications for using video recordings in SR, as they provide an accurate and comprehensive depiction of the classroom experience, enabling a more efficient analysis of video recordings in SR. By integrating intelligent algorithms, the effectiveness of retrospection can be enhanced, as algorithm-supported stimuli playback offers a reflective cognitive scaffolding beyond the mere recollection of the learning process.

Considering that many researchers have begun to incorporate physiological data as a source of stimuli (14 articles, as indicated in Table 3 ), the application of computer vision or machine learning algorithms could also be instrumental in capturing and analyzing learners’ physiological data in a lightweight manner, such as recognizing and analyzing their gestures and micro-expressions via webcam, which enriches the informativeness of stimuli (Zhai et al., 2022 ). Machine learning algorithms can now identify and analyze patterns in learner behavior that may not be apparent to human observers, providing a more nuanced understanding of cognitive processes. Moreover, intelligent algorithms enhance the reliability of findings and can also prevent the potential for the Hawthorne effect resulting from direct observation and data collection.

Facilitating the implementation interviews by using virtual agents powered by generative AI

Interviews are instrumental in the process of SR, with the majority of researchers opting to pose not merely general enquiries, but targeted ones (99 articles, as referenced in Table 3 ). This necessitated the undertaking of comprehensive interviews with each participant, a process that is notably time-intensive and requires substantial human endeavor. Future research could explore the use of virtual agents supported by generative AI technologies as an alternative approach to completing the questioning process of SR. Educational research has shown that intelligent agents positively affect learner motivation, academic performance, and cognitive load, making them ideal for training learners’ metacognitive abilities (Dinçer and Doğanay, 2017 ; Kautzmann and Jaques, 2019 ).

Intelligent conversational agents powered by natural language processing (NLP) and large language models (LLM) can replace researchers in providing participants with questions that prompt their recall and offer adaptive feedback based on their responses (Bozkurt, 2023 ). Employing intelligent agents to conduct interviews increases the number of subjects without increasing labor costs. For instance, OpenAI has developed several cutting-edge AI technologies, including the GPT series of language models such as ChatGPT, which can presumably be applied to provide personalized intelligent tutoring services in which feedback-enabled iterative learning occurs (Qadir, 2022 ). Furthermore, the LangChain architecture makes it easier to develop domain-specific agents. Such technologies provide tailored feedback to learners, enhancing their metacognitive awareness and learning outcomes. Additionally, recent advancements in generative AI have shown promising results in producing various forms of multimedia content, including text, images, videos, and 3D models (Gozalo-Brizuela and Garrido-Merchan, 2023 ). This ability to generate multimodal content aligns with the Cognitive Theory of Multimedia Learning, which emphasizes using multiple sensory channels to facilitate learning experiences (Mayer, 2002 ; Mayer and Moreno, 2003 ). By providing learners with diverse visual and auditory information, this technology can enhance the effectiveness of educational activities and promote reflection among instructors and students.

Nonetheless, using virtual agents in educational SR research raises ethical and privacy concerns that require attention in future studies. Firstly, the employment of generative AI in SR interviews involves communicating students’ sensitive data, including grades or personal information. Secondly, conversational virtual agents are trained on specific data, leading to the possibility of biased and discriminatory responses when posing SR questions. Therefore, SR researchers must utilize generative AI responsibly and ethically (Mhlanga, 2023 ).

Conclusions

This study reviews 257 empirical articles on using SR in education research from 2012 to 2022. The paper examines the changes and adaptations of the SR method in the present educational landscape, where virtual and online spaces are prevalent, and technological tools are increasingly involved in the teaching and learning process.

The revealed that researchers frequently employed SR to investigate participants’ internal viewpoints and thoughts, improving their metacognitive abilities. In terms of stimuli selection and processing, the commonly employed video stimuli undergo continuous advancements. Moreover, the sources of stimuli are becoming diverse, with the inclusion of physiological feedback data. Additionally, providing participants with space to respond to interview questions is crucial. Researchers should ensure the discussion does not deviate from the research questions and avoid influencing participants’ thoughts.

Furthermore, using technologies such as generative AI can enhance the reliability and generalizability of SR, and the study proposes suggestions for future research in result enhancement, stimuli optimization, and interview implementation. This study provides theoretical supplementation to manifesting the Retrocue Effect in educational settings. It strengthens the Cognitive Theory of Multimedia Learning (CTML) with specific pedagogical strategies by combining it with SR. From a practical perspective, the current research synthesizes current findings and can serve as a valuable reference for educators and researchers in this field.

As with any systematic review, the current research has limitations inherent to the selection and filtering process. Primarily, the sample size is restricted to articles available through the Web of Science, IEEE, and Scopus databases. There might be relevant and high-qualify studies published outside these three databases and are worthy studying. We hope future researchers can build on our research and offer a more comprehensive review of the use of SR in education.

In addition, education has now entered the era of Metaverse and artificial intelligence (Wang et al., 2022 ). How can instructors effectively apply SR in 3D virtual learning environments and in learning setting empowered by AI and AIGC (AI-generated content) remains a new territory for our continued research.

Data availability

All data generated or analyzed during this study are included in this published article.

Agricola BT, Prins FJ, van der Schaaf MF et al. (2021) Supervisor and student perspectives on undergraduate thesis supervision in higher education. Scand J Educ Res 65(5):877–897

Article Google Scholar

Al Mamun MA, Lawrie G, Wright T (2020) Instructional design of scaffolded online learning modules for self-directed and inquiry-based learning environments. Comput Educ 144:103695

Amador JM, Bragelman J, Superfine AC (2021) Prospective teachers’ noticing: a literature review of methodological approaches to support and analyze noticing. Teach Teach Educ 99:103256

Belvis E, Pineda P, Armengol C et al. (2013) Evaluation of reflective practice in teacher education. Eur J Teach Educ 36(3):279–292

Bloom BS (1953) Thought-processes in lectures and discussions. J Gen Educ 7(3):160–169

Google Scholar

Bogard T, Liu M, Chiang YHV (2013) Thresholds of knowledge development in complex problem solving: a multiple-case study of advanced learners’ cognitive processes. Educ Tech Res Dev 61(3):465–503

Bowles MA (2018) Introspective verbal reports: think-alouds and stimulated recall. In: Phakiti A, De Costa P, Plonsky L et al. (eds.) The Palgrave handbook of applied linguistics research methodology. Palgrave Macmillan, London, p 423-457

Bozkurt A (2023) Generative artificial intelligence (AI) powered conversational educational agents: the inevitable paradigm shift. Asian J Distance Educ 18(1):198–204

Calderhead J (1981) Stimulated recall: a method for research on teaching. Br J Educ Psychol 51(2):211–217

Cao Z, Yu S, Huang J (2019) A qualitative inquiry into undergraduates’ learning from giving and receiving peer feedback in L2 writing: Insights from a case study. Stud Educ Eval 63:102–112

Chan KKH, Yung BHW (2015) On-site pedagogical content knowledge development. Int J Sci Educ 37(8):1246–1278

Chan KKH, Xu L, Cooper R et al. (2021) Teacher noticing in science education: do you see what I see? Stud Sci Educ 57(1):1–44

Chu X, Zhai X (2023) A Systematic Review of Stimulated Recall (SR) in Education from 2012 to 2021. In: Shih JL et al. (eds.), Main Conference Proceedings (English Paper) of the 27th Global Chinese Conference on Computers in Education (GCCCE 2023) (pp. 100-108). China: Beijing Normal University. https://aic-fe.bnu.edu.cn/fj/lunwenji2023/EnglishPaper.pdf

Consuegra E, Engels N, Willegems V (2016) Using video-stimulated recall to investigate teacher awareness of explicit and implicit gendered thoughts on classroom interactions. Teach Teach 22(6):683–699

D’Oca G, Hrynaszkiewicz I (2015) Palgrave Communications’ commitment to promoting transparency and reproducibility in research. Palgr Commun. 1(1):1–3

Deng L (2020) Laptops and mobile phones at self-study time: examining the mechanism behind interruption and multitasking. Australas J Educ Technol 36(1):55–67

Dewey J (1933) How we think: a restatement of the relation of reflective thinking to the educative process. DC Heath

Dinçer S, Doğanay A (2017) The effects of multiple-pedagogical agents on learners’ academic success, motivation, and cognitive load. Comput Educ 111:74–100

dos Santos RAT (2018) Ways of using musical knowledge to think about one’s piano repertoire learning: three case studies. Music Educ Res 20(4):427–445

Dos Santos RAT, Hentschke L (2011) Praxis and poiesis in piano repertoire preparation. Music Educ Res 13(3):273–292

dos Santos S, Loveridge J (2014) Using video to promote early childhood teachers’ thinking and reflection. Teach Teach Educ 41:42–51

Duarte A, Hearons P, Jiang Y et al. (2013) Retrospective attention enhances visual working memory in the young but not the old: an ERP study. Psychophysiology 50(5):465–476

Article PubMed PubMed Central Google Scholar

Duo S, Song LX (2012) An e-learning system based on affective computing. Phys Procedia 24:1893–1898

Article ADS Google Scholar

Egi T (2008) Investigating stimulated recall as a cognitive measure: Reactivity and verbal reports in SLA research methodology. Lang Aware 17(3):212–228

El E, Windeatt S (2019) Eye tracking analysis of EAP Students’ regions of interest in computer-based feedback on grammar, usage, mechanics, style and organization and development. System 83:36–49

Endacott JL (2016) Using video-stimulated recall to enhance preservice-teacher reflection. N Educ 12(1):28–47

Farahian M (2015) Assessing EFL learners’ writing metacognitive awareness. J Lang Linguist Stud 11(2):39–51

Fernandez CJ (2018) Behind a spoken performance: test takers’ strategic reactions in a simulated part 3 of the IELTS speaking test. Lang Test Asia 8(1):18

Flavell JH (1979) Metacognition and cognitive monitoring:a new area of cognitive–developmental inquiry. Am Psychol 34(10):906

Gass SM, Mackey A (2000) Stimulated recall methodology in second language research. Routledge, New York

Gass SM, Mackey A (2016) Stimulated recall methodology in applied linguistics and L2 research. Routledge, New York

Book Google Scholar

Gazdag E, Nagy K, Szivák J (2019) “I Spy with My Little Eyes…” The use of video stimulated recall methodology in teacher training–The exploration of aims, goals and methodological characteristics of VSR methodology through systematic literature review. Int J Educ Res 95:60–75

Geiger V, Muir T, Lamb J (2016) Video-stimulated recall as a catalyst for teacher professional learning. J Math Teach Educ 19:457–475

Gijselaers HJ, Kirschner PA, Verboon P et al. (2016) Sedentary behavior and not physical activity predicts study progress in distance education. Learn Individ Differ 49:224–229

Goldberg P, Sümer Ö, Stürmer K et al. (2021) Attentive or not? Toward a machine learning approach to assessing students’ visible engagement in classroom instruction. Educ Psychol Rev 33:27–49

Golhasany H, Harvey B (2023) Capacity development for knowledge mobilization: a scoping review of the concepts and practices. Hum Soc Sci Commun 10(1):1–12

Gozalo-Brizuela R, Garrido-Merchan EC (2023) ChatGPT is not all you need. A State-of-the-Art Review of large Generative AI models. arXiv. https://arxiv.org/abs/2301.04655

Harvey W, Wilkinson S, Pressé C et al. (2014) Children say the darndest things: physical activity and children with attention-deficit hyperactivity disorder. Phys Educ Sport Pedag 19(2):205–220

Heikonen L, Toom A, Pyhältö K et al. (2017) Student-teachers’ strategies in classroom interaction in the context of the teaching practicum. J Educ Teach 43(5):534–549

Hu J, Gao X (2020) Appropriation of resources by bilingual students for self-regulated learning of science. Int J Biling Educ Biling 23(5):567–583

Hu J, Wu P (2020) Understanding English language learning in tertiary English-medium instruction contexts in China. System 93:102305

Jackson DO, Cho M (2018) Language teacher noticing: a socio-cognitive window on classroom realities. Lang Teach Res 22(1):29–46

Jacobson MJ, Wilensky U (2006) Complex systems in education: scientific and educational importance and implications for the learning sciences. J Learn Sci 15(1):11–34

Jacobson MJ, Levin JA, Kapur M (2019) Education as a complex system: conceptual and methodological implications. Educ Res 48(2):112–119

Jensen A (2019) Fostering preservice teacher agency in 21st century writing instruction. Engl Teach Pract Crit 18(3):298–311

Kang YS, Pyun DO (2013) Mediation strategies in L2 writing processes: a case study of two Korean language learners. Lang Cult Curric 26(1):52–67

Karimi MN, Norouzi M (2017) Scaffolding teacher cognition: changes in novice L2 teachers’ pedagogical knowledge base through expert mentoring initiatives. System 65:38–48

Kautzmann TR, Jaques PA (2019) Effects of adaptive training on metacognitive knowledge monitoring ability in computer-based learning. Comput Educ 129:92–105

Keith MJ (1988) Stimulated recall and teachers’ thought processes: a critical review of the methodology and an alternative perspective. Paper presented at the 17th Annual Meeting of the Mid-South Educational Research Association, Louisville, KY, 9−11

Koltovskaia S (2020) Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: a multiple case study. Assess 44:100450

Kurki K, Järvenoja H, Järvelä S et al. (2016) How teachers co-regulate children’s emotions and behaviour in socio-emotionally challenging situations in day-care settings. Int J Educ Res 76:76–88

Lai ML, Tsai MJ, Yang FY et al. (2013) A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educ Res Rev 10:90–115

Lambert C, Zhang G (2019) Engagement in the use of English and Chinese as foreign languages: the role of learner‐generated content in instructional task design. Mod Lang J 103(2):391–411

Lee C (2020) A study of adolescent English learners’ cognitive engagement in writing while using an automated content feedback system. Comput Assist Lang. Learn 33(1-2):26–57

Lichtinger E, Kaplan A (2015) Employing a case study approach to capture motivation and self-regulation of young students with learning disabilities in authentic educational contexts. Metacogn Learn 10(1):119–149

Lin CH, Zhang Y, Zheng B (2017) The roles of learning strategies and motivation in online language learning: a structural equation modeling analysis. Comput Educ 113:75–85

Lindfors M, Bodin M, Simon S (2020) Unpacking students’ epistemic cognition in a physics problem‐solving environment. J Res Sci Teach 57(5):695–732

Łucznik K, May J (2021) Measuring individual and group flow in collaborative improvisational dance. Think Skills Creat 40:100847

Lyle J (2003) Stimulated recall: a report on its use in naturalistic research. Br Educ Res J 29(6):861–878

Maaranen K, Kynäslahti H, Byman R et al. (2019) Teacher education matters: finnish teacher educators’ concerns, beliefs, and values. Eur J Teach Educ 42(2):211–227

Määttä E, Mykkänen A, Järvelä S (2016) Elementary schoolchildrenas self-and social perceptions of success. J Res Child Educ 30(2):170–184

Martinelle R (2018) Video-stimulated recall: aiding teacher practice. Educ Leadersh 76(3):55

Martinelle R (2020) Using video-stimulated recall to understand the reflections of ambitious social studies teachers. J Soc Stud Res 44(3):307–322

Mayer RE, Moreno R (1998) A cognitive theory of multimedia learning: Implications for design principles. J Educ Psychol 91(2):358–368

Mayer RE, Moreno R (2003) Nine ways to reduce cognitive load in multimedia learning. Educ Psychol 38(1):43–52

Mayer RE (2002) Multimedia learning. In: Brian HR (ed.) Psychology of learning and motivation, vol 41. Academic Press, Cambridge, p 85-139

Meade P, McMeniman M (1992) Stimulated recall—An effective methodology for examining successful teaching in science. Aust Educ Res 19(3):1–18

Mhlanga D (2023) Open AI in Education, the Responsible and Ethical Use of ChatGPT Towards Lifelong Learning. In: Mhlanga D (ed.) FinTech and Artificial Intelligence for Sustainable Development. Sustainable Development Goals Series. Palgrave Macmillan, Cham, pp 387-409

Michel M, Révész A, Lu X et al. (2020) Investigating L2 writing processes across independent and integrated tasks: a mixed-methods study. Second Lang Res 36(3):307–334

Moher D, Liberati A, Tetzlaff J et al. (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 151(4):264–269

Article PubMed Google Scholar

Molenaar I, Knoop-van Campen CA (2018) How teachers make dashboard information actionable. IEEE Trans Learn Technol 12(3):347–355

Moreno R, Mayer RE (1999) Cognitive principles of multimedia learning: the role of modality and contiguity. J Educ Psychol 91(2):358

Myrtil MJ, Lin TJ, Chen J et al. (2021) Pros and (con) flict: using head-mounted cameras to identify teachers’ roles in intervening in conflict among preschool children. Early Child Res Q 55:230–241

Nguyen NT, McFadden A, Tangen D et al. (2013) Video-stimulated recall interviews in qualitative research. Paper presented at the Annual Meeting of the Australian Association for Research in Education (AARE), Adelaide, Australia, 1-5 December 2013

Nurmukhamedov U, Kim SH (2010) ‘Would you perhaps consider…’: hedged comments in ESL writing. ELT J 64(3):272–282

Nyberg G, Larsson H (2017) Physical education teachers’ content knowledge of movement capability. J Teach Phys Educ 36(1):61–69

Obersteiner A, Tumpek C (2016) Measuring fraction comparison strategies with eye-tracking. ZDM 48(3):255–266

Page MJ, McKenzie JE, Bossuyt PM et al. (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg 88:105906

Park M (2018) Innovative assessment of aviation English in a virtual world: windows into cognitive and metacognitive strategies. ReCALL 30(2):196–213

Peng JE (2020) Teacher interaction strategies and situated willingness to communicate. ELT J 74(3):307–317

Perkins R, Aufegger L, Williamon A (2015) Learning through teaching: exploring what conservatoire students learn from teaching beginner older adults. Int J Music Educ 33(1):80–90

Pratt SM, Martin AM (2017) The differential impact of video-stimulated recall and concurrent questioning methods on beginning readers’ verbalization about self-monitoring during oral reading. Read Psychol 38(5):439–485

Qadir J (2022) Engineering education in the era of ChatGPT: promise and pitfalls of generative AI for education. In: 2023 IEEE Global Engineering Education Conference (EDUCON), Kuwait, Kuwait, 1-4 May 2023

Qiu X (2020) Functions of oral monologic tasks: effects of topic familiarity on L2 speaking performance. Lang Teach Res 24(6):745–764

Qiu X, Lo YY (2017) Content familiarity task repetition and Chinese EFL learners’ engagement in second language use. Lang Teach Res 21(6):681–698

Ramadhanti D, Yanda DP (2021) Students’ metacognitive awareness and its impact on writing skill. Int J Lang Educ 5(3):193–206

Ramakrishnan A, Zylich B, Ottmar E et al. (2023) Toward automated classroom observation: multimodal machine learning to estimate class positive climate and negative climate. IEEE Trans Affect Comput 14(1):664–679

Ramnarain UD, Modiba M (2013) Critical friendship collaboration and trust as a basis for self-determined professional development: a case of science teaching. Int J Sci Educ 35(1):65–85

Rankanen M, Leinikka M, Groth C et al. (2022) Physiological measurements and emotional experiences of drawing and clay forming. Arts Psychother 79:101899

Rassaei E (2013) Corrective feedback, learners’ perceptions, and second language development. System 41(2):472–483

Rassaei E (2015) Journal writing as a means of enhancing EFL learners’ awareness and effectiveness of recasts. Linguist Educ 32:118–130

Rassaei E (2020) The separate and combined effects of recasts and textual enhancement as two focus on form techniques on L2 development. System 89:102193

Révész A, Kourtali NE, Mazgutova D (2017) Effects of task complexity on L2 writing behaviors and linguistic complexity. Lang Learn 67(1):208–241

Rietdijk S, van Weijen D, Janssen T et al. (2018) Teaching writing in primary education: classroom practice, time, teachers’ beliefs and skills. J Educ Psychol 110(5):640

Rissanen I, Kuusisto E, Hanhimäki E et al. (2018) The implications of teachers’ implicit theories for moral education: a case study from Finland. J Moral Educ 47(1):63–77

Rissanen I, Kuusisto E, Tuominen M et al. (2019) In search of a growth mindset pedagogy: a case study of one teacher’s classroom practices in a Finnish elementary school. Teach Teach Educ 77:204–213

Sanchez HS, Grimshaw T (2019) Stimulated recall. In: McKinley J, Rose H (eds.) The Routledge handbook of research methods in applied linguistics. Routledge, Abingdon, pp 312-323

Sato R (2019) Fluctuations in an EFL teacher’s willingness to communicate in an English-medium lesson: an observational case study in Japan. Innov Lang Learn Teach 13(2):105–117

Schindler M, Lilienthal AJ (2019) Domain-specific interpretation of eye tracking data: towards a refined use of the eye-mind hypothesis for the field of geometry. Educ Stud Math 101(1):123–139

Schindler M, Lilienthal AJ (2020) Students’ creative process in mathematics: insights from eye-tracking-stimulated recall interview on students’ work on multiple solution tasks. Int J Sci Math Educ 18(8):1565–1586

Schneider D, Barth A, Getzmann S et al. (2017) On the neural mechanisms underlying the protective function of retroactive cuing against perceptual interference: evidence by event-related potentials of the EEG. Biol Psychol 124:47–56

Shepherdson P, Oberauer K, Souza AS (2018) Working memory load and the retro-cue effect: a diffusion model account. J Exp Psychol Hum Percept Perform 44(2):286

Shintani N (2016) The effects of computer-mediated synchronous and asynchronous direct corrective feedback on writing: a case study. Comput Assist Lang Learn 29(3):517–538

Smagorinsky P (1998) Thinking and speech and protocol analysis. Mind Cult Act 5(3):157–177

Smithers LG, Sawyer AC, Chittleborough CR et al. (2018) A systematic review and meta-analysis of effects of early life non-cognitive skills on academic, psychosocial, cognitive and health outcomes. Nat Hum Behav 2(11):867–880

Souza AS, Oberauer K (2016) In search of the focus of attention in working memory: 13 years of the retro-cue effect. Atten Percept Psycho 78:1839–1860

Stolpe K, Björklund L (2013) Students’ long-term memories from an ecology field excursion: retelling a narrative as an interplay between implicit and explicit memories. Scand J Educ Res 57(3):277–291

Stough LM (2001) Using Stimulated Recall in Classroom Observation and Professional Development. Paper presented at the Annual Meeting of the American. Educational Research Association, Seattle, WA, 10-14 April 2001

Sundberg B, Areljung S, Due K et al. (2018) Opportunities for and obstacles to science in preschools: views from a community perspective. Int J Sci Educ 40(17):2061–2077

Tan ST, Tan CX, Tan SS (2021) Physical activity, sedentary behavior, and weight status of university students during the covid-19 lockdown: a cross-national comparative study. Int J Environ Res Public Health 18(13):7125

Article CAS PubMed PubMed Central Google Scholar

Thararuedee P, Wette R (2020) Attending to learners’ affective needs: teachers’ knowledge and practices. System 95:102375

Tiainen O, Korkeamäki RL, Dreher MJ (2018) Becoming reflective practitioners: a case study of three beginning pre-service teachers. Scand J Educ Res 62(4):586–600

Ucan S, Webb M (2015) Social regulation of learning during collaborative inquiry learning in science: how does it emerge and what are its functions? Int J Sci Educ 37(15):2503–2532

Van der Kleij F, Adie L, Cumming J (2017) Using video technology to enable student voice in assessment feedback. Br J Educ Technol 48(5):1092–1105

Walan S, Enochsson AB (2019) The potential of using a combination of storytelling and drama, when teaching young children science. Eur Early Child Educ Res J 27(6):821–836

Wang M, Yu H, Bell Z et al. (2022) Constructing an Edu-Metaverse ecosystem: a New and innovative framework. IEEE Trans Learn Technol 15(6):685–696

Wass R, Moskal ACM (2017) What can Interpersonal Process Recall (IPR) offer academic development? Int J Acad Dev 22(4):293–306

White C, Direnzo R, Bortolotto C (2016) The learner-context interface: emergent issues of affect and identity in technology-mediated language learning spaces. System 62:3–14

Wijayasundara M (2020) Integration of ICT in teaching and learning in schools. Int J Res 1(10):198–209

Wilby KJ, Govaerts MJ, Austin Z et al. (2017) Exploring the influence of cultural orientations on assessment of communication behaviours during patient-practitioner interactions. BMC Med Educ 17(1):1–10

Wu SC (2019) Researching children’s learning and play in a Chinese context: children’s perspectives on their play and learning. Eur Early Child Educ Res J 27(4):551–565

Yu S (2021) Giving genre-based peer feedback in academic writing: sources of knowledge and skills, difficulties and challenges. Assess Eval High Educ 46(1):36–53

Yu S, Hu G (2017) Understanding university students’ peer feedback practices in EFL writing: Insights from a case study. Assess Writ 33:25–35

Zahid M, Khanam A (2019) Effect of reflective teaching practices on the performance of prospective teachers. Turk Online J Educ Technol 18(1):32–43

Zhai X, Fang Q, Dong Y et al. (2018) The effects of biofeedback‐based stimulated recall on self‐regulated online learning: a gender and cognitive taxonomy perspective. J Comput Assist Learn 34(6):775–786

Zhai X, Chu X, Meng N et al. (2022) The effect of multi-mode stimuli of feedforward and eye tracking on metacognition—an exploratory study using digital dictionaries. Educ Technol Soc 25(1):213–227

Zhai X, Xu J, Chen NS et al. (2022) The Syncretic Effect of Dual-Source Data on Affective Computing in Online Learning Contexts: A Perspective From Convolutional Neural Network With Attention Mechanism. J Educ Comput Res 61(2):466–493

Download references

Acknowledgements

This research is funded by the National Science and Technology Major Project (Grant No. 2022ZD0115904), 2021 National Natural Science Foundation of China (Grant No. 62177042), and 2024 Zhejiang Provincial Natural Science Foundation of China (Grant No. Y24F020039).

Author information

These authors contributed equally: Xuesong Zhai, Xiaoyan Chu.

Authors and Affiliations

College of Education, Zhejiang University, Hangzhou, China

Xuesong Zhai & Xiaoyan Chu

Hangzhou International Urbanology Research Center & Zhejiang Urban Governance Studies Center, Hangzhou, China

Xuesong Zhai

Department of Mathematics and Information Technology, The Education University of Hong Kong, Hong Kong SAR, China

Minjuan Wang

Learning Design and Technology, San Diego State University, San Diego, CA, USA

Program of Learning Sciences and Institute for Research Excellence in Learning Sciences, National Taiwan Normal University, Taipei, Taiwan

Chin-Chung Tsai & Jyh-Chong Liang

Department of Learning Technologies, University of North Texas, Denton, TX, USA

Jonathan Michael Spector

You can also search for this author in PubMed Google Scholar

Contributions

All authors have contributed to material preparation and data analysis. Conceptualization, design and data collection: XS Zhai, and XY Chu. Methodology: All. Original draft: XS Zhai and XY Chu. Second draft: MJ Wang. Third draft: CC Tsai and JC Liang. Final round revision and quality check: JM Spector. All authors discussed the results and reviewed, edited, and approved the final version of the manuscript.

Corresponding authors

Correspondence to Xiaoyan Chu or Minjuan Wang .

Ethics declarations

Competing interests.

The author declares no competing interests.

Ethical approval

Ethical approval was not required as the study did not involve human participants.

Informed consent

This article does not contain any studies with human participants performed by any of the authors.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Zhai, X., Chu, X., Wang, M. et al. A systematic review of Stimulated Recall (SR) in educational research from 2012 to 2022. Humanit Soc Sci Commun 11 , 489 (2024). https://doi.org/10.1057/s41599-024-02987-6

Download citation

Received : 01 June 2023

Accepted : 25 March 2024

Published : 05 April 2024

DOI : https://doi.org/10.1057/s41599-024-02987-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

Explore articles by subject
Guide to authors
Editorial policies

Open access
Published: 02 April 2022

A qualitative study of rural healthcare providers’ views of social, cultural, and programmatic barriers to healthcare access

Nicholas C. Coombs 1 ,
Duncan G. Campbell 2 &
James Caringi 1

BMC Health Services Research volume 22 , Article number: 438 ( 2022 ) Cite this article

22k Accesses

19 Citations

10 Altmetric

Metrics details

Ensuring access to healthcare is a complex, multi-dimensional health challenge. Since the inception of the coronavirus pandemic, this challenge is more pressing. Some dimensions of access are difficult to quantify, namely characteristics that influence healthcare services to be both acceptable and appropriate. These link to a patient’s acceptance of services that they are to receive and ensuring appropriate fit between services and a patient’s specific healthcare needs. These dimensions of access are particularly evident in rural health systems where additional structural barriers make accessing healthcare more difficult. Thus, it is important to examine healthcare access barriers in rural-specific areas to understand their origin and implications for resolution.

We used qualitative methods and a convenience sample of healthcare providers who currently practice in the rural US state of Montana. Our sample included 12 healthcare providers from diverse training backgrounds and specialties. All were decision-makers in the development or revision of patients’ treatment plans. Semi-structured interviews and content analysis were used to explore barriers–appropriateness and acceptability–to healthcare access in their patient populations. Our analysis was both deductive and inductive and focused on three analytic domains: cultural considerations, patient-provider communication, and provider-provider communication. Member checks ensured credibility and trustworthiness of our findings.

Five key themes emerged from analysis: 1) a friction exists between aspects of patients’ rural identities and healthcare systems; 2) facilitating access to healthcare requires application of and respect for cultural differences; 3) communication between healthcare providers is systematically fragmented; 4) time and resource constraints disproportionately harm rural health systems; and 5) profits are prioritized over addressing barriers to healthcare access in the US.

Conclusions

Inadequate access to healthcare is an issue in the US, particularly in rural areas. Rural healthcare consumers compose a hard-to-reach patient population. Too few providers exist to meet population health needs, and fragmented communication impairs rural health systems’ ability to function. These issues exacerbate the difficulty of ensuring acceptable and appropriate delivery of healthcare services, which compound all other barriers to healthcare access for rural residents. Each dimension of access must be monitored to improve patient experiences and outcomes for rural Americans.

Peer Review reports

Unequal access to healthcare services is an important element of health disparities in the United States [ 1 ], and there remains much about access that is not fully understood. The lack of understanding is attributable, in part, to the lack of uniformity in how access is defined and evaluated, and the extent to which access is often oversimplified in research [ 2 ]. Subsequently, attempts to address population-level barriers to healthcare access are insufficient, and access remains an unresolved, complex health challenge [ 3 , 4 , 5 ]. This paper presents a study that aims to explore some of the less well studied barriers to healthcare access, particularly those that influence healthcare acceptability and appropriateness.

In truth, healthcare access entails a complicated calculus that combines characteristics of individuals, their households, and their social and physical environments with characteristics of healthcare delivery systems, organizations, and healthcare providers. For one to fully ‘access’ healthcare, they must have the means to identify their healthcare needs and have available to them care providers and the facilities where they work. Further, patients must then reach, obtain, and use the healthcare services in order to have their healthcare needs fulfilled. Levesque and colleagues critically examined access conceptualizations in 2013 and synthesized all ways in which access to healthcare was previously characterized; Levesque et al. proposed five dimensions of access: approachability, acceptability, availability, affordability and appropriateness [ 2 ]. These refer to the ability to perceive, seek, reach, pay for, and engage in services, respectively.

According to Levesque et al.’s framework, the five dimensions combine to facilitate access to care or serve as barriers. Approachability indicates that people facing health needs understand that healthcare services exist and might be helpful. Acceptability represents whether patients see healthcare services as consistent or inconsistent with their own social and cultural values and worldviews. Availability indicates that healthcare services are reached both physically and in a timely manner. Affordability simplifies one’s capacity to pay for healthcare services without compromising basic necessities, and finally, appropriateness represents the fit between healthcare services and a patient’s specific healthcare needs [ 2 ]. This study focused on the acceptability and appropriateness dimensions of access.

Before the novel coronavirus (SARS-CoV-2; COVID-19) pandemic, approximately 13.3% of adults in the US did not have a usual source of healthcare [ 6 ]. Millions more did not utilize services regularly, and close to two-thirds reported that they would be debilitated by an unexpected medical bill [ 7 , 8 , 9 ]. Findings like these emphasized a fragility in the financial security of the American population [ 10 ]. These concerns were exacerbated by the pandemic when a sudden surge in unemployment increased un- and under-insurance rates [ 11 ]. Indeed, employer-sponsored insurance covers close to half of Americans’ total cost of illness [ 12 ]. Unemployment linked to COVID-19 cut off the lone outlet to healthcare access for many. Health-related financial concerns expanded beyond individuals, as healthcare organizations were unequipped to manage a simultaneous increase in demand for specialized healthcare services and a steep drop off for routine revenue-generating healthcare services [ 13 ]. These consequences of the COVID-19 pandemic all put additional, unexpected pressure on an already fragmented US healthcare system.

Other structural barriers to healthcare access exist in relation to the rural–urban divide. Less than 10% of US healthcare resources are located in rural areas where approximately 20% of the American population resides [ 14 ]. In a country with substantially fewer providers per capita compared to many other developed countries, persons in rural areas experience uniquely pressing healthcare provider shortages [ 15 , 16 ]. Rural inhabitants also tend to have lower household income, higher rates of un- or under-insurance, and more difficulty with travel to healthcare clinics than urban dwellers [ 17 ]. Subsequently, persons in rural communities use healthcare services at lower rates, and potentially preventable hospitalizations are more prevalent [ 18 ]. This disparity often leads rural residents to use services primarily for more urgent needs and less so for routine care [ 19 , 20 , 21 ].

The differences in how rural and urban healthcare systems function warranted a federal initiative to focus exclusively on rural health priorities and serve as counterpart to Healthy People objectives [ 22 ]. The rural determinants of health, a more specific expression of general social determinants, add issues of geography and topography to the well-documented social, economic and political factors that influence all Americans’ access to healthcare [ 23 ]. As a result, access is consistently regarded as a top priority in rural areas, and many research efforts have explored the intersection between access and rurality, namely within its less understood dimensions (acceptability and appropriateness) [ 22 ].

Acceptability-related barriers to care

Acceptability represents the dimension of healthcare access that affects a patient’s ability to seek healthcare, particularly linked to one’s professional values, norms and culture [ 2 ]. Access to health information is an influential factor for acceptable healthcare and is essential to promote and maintain a healthy population [ 24 ]. According to the Centers for Disease Control and Prevention, health literacy or a high ‘health IQ’ is the degree to which individuals have the ability to find, understand, and use information and services to inform health-related decisions and actions for themselves and others, which impacts healthcare use and system navigation [ 25 ]. The literature indicates that lower levels of health literacy contribute to health disparities among rural populations [ 26 , 27 , 28 ]. Evidence points to a need for effective health communication between healthcare organizations and patients to improve health literacy [ 24 ]. However, little research has been done in this area, particularly as it relates to technologically-based interventions to disseminate health information [ 29 ].

Stigma, an undesirable position of perceived diminished status in an individual’s social position, is another challenge that influences healthcare acceptability [ 30 ]. Those who may experience stigma fear negative social consequences in relation to care seeking. They are more likely to delay seeking care, especially among ethnic minority populations [ 31 , 32 ]. Social media presents opportunities for the dissemination of misleading medical information; this runs further risk for stigma [ 33 ]. Stigma is difficult to undo, but research has shown that developing a positive relationship with a healthcare provider or organization can work to reduce stigma among patients, thus promoting healthcare acceptability [ 34 ].

A provider’s attempts to engage patients and empower them to be active decision-makers regarding their treatment has also been shown to improve healthcare acceptability. One study found that patients with heart disease who completed a daily diary of weight and self-assessment of symptoms, per correspondence with their provider, had better care outcomes than those who did not [ 35 ]. Engaging with household family members and involved community healers also mitigates barriers to care, emphasizing the importance of a team-based approach that extends beyond those who typically provide healthcare services [ 36 , 37 ]. One study, for instance, explored how individuals closest to a pregnant woman affect the woman’s decision to seek maternity care; partners, female relatives, and community health-workers were among the most influential in promoting negative views, all of which reduced a woman’s likelihood to access care [ 38 ].

Appropriateness-related barriers to care

Appropriateness marks the dimension of healthcare access that affects a patient’s ability to engage, and according to Levesque et al., is of relevance once all other dimensions (the ability to perceive, seek, reach and pay for) are achieved [ 2 ]. The ability to engage in healthcare is influenced by a patient’s level of empowerment, adherence to information, and support received by their healthcare provider. Thus, barriers to healthcare access that relate to appropriateness are often those that indicate a breakdown in communication between a patient with their healthcare provider. Such breakdown can involve a patient experiencing miscommunication, confrontation, and/or a discrepancy between their provider’s goals and their own goals for healthcare. Appropriateness represents a dimension of healthcare access that is widely acknowledged as an area in need of improvement, which indicates a need to rethink how healthcare providers and organizations can adapt to serve the healthcare needs of their communities [ 39 ]. This is especially true for rural, ethnic minority populations, which disproportionately experience an abundance of other barriers to healthcare access. Culturally appropriate care is especially important for members of minority populations [ 40 , 41 , 42 ]. Ultimately, patients value a patient-provider relationship characterized by a welcoming, non-judgmental atmosphere [ 43 , 44 ]. In rural settings especially, level of trust and familiarity are common factors that affect service utilization [ 45 ]. Evidence suggests that kind treatment by a healthcare provider who promotes patient-centered care can have a greater overall effect on a patient’s experience than a provider’s degree of medical knowledge or use of modern equipment [ 46 ]. Of course, investing the time needed to nurture close and caring interpersonal connections is particularly difficult in under-resourced, time-pressured rural health systems [ 47 , 48 ].

The most effective way to evaluate access to healthcare largely depends on which dimensions are explored. For instance, a population-based survey can be used to measure the barrier of healthcare affordability. Survey questions can inquire directly about health insurance coverage, care-related financial burden, concern about healthcare costs, and the feared financial impacts of illness and/or disability. Many national organizations have employed such surveys to measure affordability-related barriers to healthcare. For example, a question may ask explicitly about financial concerns: ‘If you get sick or have an accident, how worried are you that you will not be able to pay your medical bills?’ [ 49 ]. Approachability and availability dimensions of access are also studied using quantitative analysis of survey questions, such as ‘Is there a place that you usually go to when you are sick or need advice about your health?’ or ‘Have you ever delayed getting medical care because you couldn’t get through on the telephone?’ In contrast, the remaining two dimensions–acceptability and appropriateness–require a qualitative approach, as the social and cultural factors that determine a patient’s likelihood of accepting aspects of the services that are to be received (acceptability) and the fit between those services and the patient’s specific healthcare needs (appropriateness) can be more abstract [ 50 , 51 ]. In social science, qualitative methods are appropriate to generate knowledge of what social events mean to individuals and how those individuals interact within them; these methods allow for an exploration of depth rather than breadth [ 52 , 53 ]. Qualitative methods, therefore, are appropriate tools for understanding the depth of healthcare providers’ experiences in the inherently social context of seeking and engaging in healthcare.

In sum, acceptability- and appropriateness-related barriers to healthcare access are multi-layered, complex and abundant. Ensuring access becomes even more challenging if structural barriers to access are factored in. In this study, we aimed to explore barriers to healthcare access among persons in Montana, a historically underserved, under-resourced, rural region of the US. Montana is the fourth largest and third least densely populated state in the country; more than 80% of Montana counties are classified as non-core (the lowest level of urban/rural classification), and over 90% are designated as health professional shortage areas [ 54 , 55 ]. Qualitative methods supported our inquiry to explore barriers to healthcare access related to acceptability and appropriateness.

Participants

Qualitative methods were utilized for this interpretive, exploratory study because knowledge regarding barriers to healthcare access within Montana’s rural health systems is limited. We chose Montana healthcare providers, rather than patients, as the population of interest so we may explore barriers to healthcare access from the perspective of those who serve many persons in rural settings. Inclusion criteria required study participants to provide direct healthcare to patients at least one-half of their time. We defined ‘provider’ as a healthcare organization employee with clinical decision-making power and the qualifications to develop or revise patients’ treatment plans. In an attempt to capture a group of providers with diverse experience, we included providers across several types and specialties. These included advanced practice registered nurses (APRNs), physicians (MDs and DOs), and physician assistants (PAs) who worked in critical care medicine, emergency medicine, family medicine, hospital medicine, internal medicine, pain medicine, palliative medicine, pediatrics, psychiatry, and urgent care medicine. We also included licensed clinical social workers (LCSWs) and clinical psychologists who specialize in behavioral healthcare provision.

Recruitment and Data Collection

We recruited participants via email using a snowball sampling approach [ 56 ]. We opted for this approach because of its effectiveness in time-pressured contexts, such as the COVID-19 pandemic, which has made healthcare provider populations hard to reach [ 57 ]. Considering additional constraints with the pandemic and the rural nature of Montana, interviews were administered virtually via Zoom video or telephone conferencing with Zoom’s audio recording function enabled. All interviews were conducted by the first author between January and September 2021. The average length of interviews was 50 min, ranging from 35 to 70 min. There were occasional challenges experienced during interviews (poor cell phone reception from participants, dropped calls), in which case the interviewer remained on the line until adequate communication was resumed. All interviews were included for analysis and transcribed verbatim into NVivo Version 12 software. All qualitative data were saved and stored on a password-protected University of Montana server. Hard-copy field notes were securely stored in a locked office on the university’s main campus.

Data analysis included a deductive followed by an inductive approach. This dual analysis adheres to Levesque’s framework for qualitative methods, which is discussed in the Definition of Analytic Domains sub-section below. Original synthesis of the literature informed the development of our initial deductive codebook. The deductive approach was derived from a theory-driven hypothesis, which consisted of synthesizing previous research findings regarding acceptability- and appropriateness-related barriers to care. Although the locations, patient populations and specific type of healthcare services varied by study in the existing literature, several recurring barriers to healthcare access were identified. We then operationalized three analytic domains based on these findings: cultural considerations, patient-provider communication, and provider-provider communication. These domains were chosen for two reasons: 1) the terms ‘culture’ and ‘communication’ were the most frequently documented characteristics across the studies examined, and 2) they each align closely with the acceptability and appropriateness dimensions of access to healthcare, respectively. In addition, ‘culture’ is included in the definition of acceptability and ‘communication’ is a quintessential aspect of appropriateness. These domains guided the deductive portion of our analysis, which facilitated the development of an interview guide used for this study.

Interviews were semi-structured to allow broad interpretations from participants and expand the open-ended characterization of study findings. Data were analyzed through a flexible coding approach proposed by Deterding and Waters [ 58 ]. Qualitative content analysis was used, a method particularly beneficial for analyzing large amounts of qualitative data collected through interviews that offers possibility of quantifying categories to identify emerging themes [ 52 , 59 ]. After fifty percent of data were analyzed, we used an inductive approach as a formative check and repeated until data saturation, or the point at which no new information was gathered in interviews [ 60 ]. At each point of inductive analysis, interview questions were added, removed, or revised in consideration of findings gathered [ 61 ]. The Standards for Reporting Qualitative Research (SRQR) was used for reporting all qualitative data for this study [ 62 ]. The first and third authors served as primary and secondary analysts of the qualitative data and collaborated to triangulate these findings. An audit approach was employed, which consisted of coding completed by the first author and then reviewed by the third author. After analyses were complete, member checks ensured credibility and trustworthiness of findings [ 63 ]. Member checks consisted of contacting each study participant to explain the study’s findings; one-third of participants responded and confirmed all findings. All study procedures were reviewed and approved by the Human Subjects Committee of the authors’ institution’s Institutional Review Board.

Definitions of Analytic Domains

Cultural considerations.

Western health systems often fail to consider aspects of patients’ cultural perspectives and histories. This can manifest in the form of a providers’ lack of cultural humility. Cultural humility is a process of preventing imposition of one’s worldview and cultural beliefs on others and recognizing that everyone’s conception of the world is valid. Humility cultivates sensitive approaches in treating patients [ 64 ]. A lack of cultural humility impedes the delivery of acceptable and appropriate healthcare [ 65 ], which can involve low empathy or respect for patients, or dismissal of culture and traditions as superstitions that interfere with standard treatments [ 66 , 67 ]. Ensuring cultural humility among all healthcare employees is a step toward optimal healthcare delivery. Cultural humility is often accomplished through training that can be tailored to particular cultural- or gender-specific populations [ 68 , 69 ]. Since cultural identities and humility have been marked as factors that can heavily influence patients’ access to care, cultural considerations composed our first analytic domain. To assess this domain, we asked participants how they address the unique needs of their patients, how they react when they observe a cultural behavior or attitude from a patient that may not directly align with their treatment plan, and if they have received any multicultural training or training on cultural considerations in their current role.

Patient-provider communication

Other barriers to healthcare access can be linked to ineffective patient-provider communication. Patients who do not feel involved in healthcare decisions are less likely to adhere to treatment recommendations [ 70 ]. Patients who experience communication difficulties with providers may feel coerced, which generates disempowerment and leads patients to employ more covert ways of engagement [ 71 , 72 ]. Language barriers can further compromise communication and hinder outcomes or patient progress [ 73 , 74 ]. Any miscommunication between a patient and provider can affect one’s access to healthcare, namely affecting appropriateness-related barriers. For these reasons, patient-provider communication composed our second analytic domain. We asked participants to highlight the challenges they experience when communicating with their patients, how those complications are addressed, and how communication strategies inform confidentiality in their practice. Confidentiality is a core ethical principle in healthcare, especially in rural areas that have smaller, interconnected patient populations [ 75 ].

Provider-Provider Communication

A patient’s journey through the healthcare system necessitates sufficient correspondence between patients, primary, and secondary providers after discharge and care encounters [ 76 ]. Inter-provider and patient-provider communication are areas of healthcare that are acknowledged to have some gaps. Inconsistent mechanisms for follow up communication with patients in primary care have been documented and emphasized as a concern among those with chronic illness who require close monitoring [ 68 , 77 ]. Similar inconsistencies exist between providers, which can lead to unclear care goals, extended hospital stays, and increased medical costs [ 78 ]. For these reasons, provider-provider communication composed our third analytic domain. We asked participants to describe the approaches they take to streamline communication after a patient’s hospital visit, the methods they use to ensure collaborative communication between primary or secondary providers, and where communication challenges exist.

Healthcare provider characteristics

Our sample included 12 providers: four in family medicine (1 MD, 1 DO, 1 PA & 1 APRN), three in pediatrics (2 MD with specialty in hospital medicine & 1 DO), three in palliative medicine (2 MDs & 1 APRN with specialty in wound care), one in critical care medicine (DO with specialty in pediatric pulmonology) and one in behavioral health (1 LCSW with specialty in trauma). Our participants averaged 9 years (range 2–15) as a healthcare provider; most reported more than 5 years in their current professional role. The diversity of participants extended to their patient populations as well, with each participant reporting a unique distribution of age, race and level of medical complexity among their patients. Most participants reported that a portion of their patients travel up to five hours, sometimes across county- or state-lines, to receive care.

Theme 1: A friction exists between aspects of patients’ rural identities and healthcare systems

Our participants comprised a collection of medical professions and reported variability among health-related reasons their patients seek care. However, most participants acknowledged similar characteristics that influence their patients’ challenges to healthcare access. These identified factors formed categories from which the first theme emerged. There exists a great deal of ‘rugged individualism’ among Montanans, which reflects a self-sufficient and self-reliant way of life. Stoicism marked a primary factor to characterize this quality. One participant explained:

True Montanans are difficult to treat medically because they tend to be a tough group. They don’t see doctors. They don’t want to go, and they don’t want to be sick. That’s an aspect of Montana that makes health culture a little bit difficult.

Another participant echoed this finding by stating:

The backwoods Montana range guy who has an identity of being strong and independent probably doesn’t seek out a lot of medical care or take a lot of medications. Their sense of vitality, independence and identity really come from being able to take care and rely on themselves. When that is threatened, that’s going to create a unique experience of illness.

Similar responses were shared by all twelve participants; stoicism seemed to be heavily embedded in many patient populations in Montana and serves as a key determinant of healthcare acceptability. There are additional factors, however, that may interact with stoicism but are multiply determined. Stigma is an example of this, presented in this context as one’s concern about judgement by the healthcare system. Respondents were openly critical of this perception of the healthcare system as it was widely discussed in interviews. One participant stated:

There is a real perception of a punitive nature in the medical community, particularly if I observe a health issue other than the primary reason for one’s hospital visit, whether that may be predicated on medical neglect, delay of care, or something that may warrant a report to social services. For many of the patients and families I see, it’s not a positive experience and one that is sometimes an uphill barrier that I work hard to circumnavigate.

Analysis of these factors suggest that low use of healthcare services may link to several characteristics, including access problems. Separately, a patient’s perceived stigma from healthcare providers may also impact a patient’s willingness to receive services. One participant put it best by stating

Sometimes, families assume that I didn’t want to see them because they will come in for follow up to meet with me but end up meeting with another provider, which is frustrating because I want to maintain patients on my panel but available time and resource occasionally limits me from doing so. It could be really hard adapting to those needs on the fly, but it’s an honest miss.

When a patient arrives for a healthcare visit and experiences this frustration, it may elicit a patient’s perceptions of neglect or disorganization. This ‘honest miss’ may, in turn, exacerbate other acceptable-related barriers to care.

Theme 2: Facilitating access to healthcare requires application of and respect for cultural differences

The biomedical model is the standard of care utilized in Western medicine [ 79 , 80 ]. However, the US comprises people with diverse social and cultural identities that may not directly align with Western conceptions of health and wellness. Approximately 11.5% of the Montana population falls within an ethnic minority group. 6.4% are of American Indian or Alaska Native origin, 0.5% are of Black or African American origin, 0.8% are of Asian origin and 3.8% are of multiple or other origins. [ 81 ]. Cultural insensitivity is acknowledged in health services research as an active deterrent for appropriate healthcare delivery [ 65 ]. Participants for this study were asked how they react when a patient brings up a cultural attitude or behavior that may impact the proposed treatment plan. Eight participants noted a necessity for humility when this occurs. One participant conceptualized this by stating:

When this happens, I learn about individuals and a way of life that is different to the way I grew up. There is a lot of beauty and health in a non-patriarchal, non-dominating, non-sexist framework, and when we can engage in such, it is really expansive for my own learning process.

The participants who expressed humility emphasized that it is best to work in tandem with their patient, congruently. Especially for those with contrasting worldviews, a provider and a patient working as a team poses an opportunity to develop trust. Without it, a patient can easily fall out of the system, further hindering their ability to access healthcare services in the future. One participant stated:

The approach that ends up being successful for a lot of patients is when we understand their modalities, and they have a sense we understand those things. We have to show understanding and they have to trust. From there, we can make recommendations to help get them there, not decisions for them to obey, rather views based on our experiences and understanding of medicine.

Curiosity was another reaction noted by a handful of participants. One participant said:

I believe patients and their caregivers can be engaged and loving in different ways that don’t always follow the prescribed approach in the ways I’ve been trained, but that doesn’t necessarily mean that they are detrimental. I love what I do, and I love learning new things or new approaches, but I also love being surprised. My style of medicine is not to predict peoples’ lives, rather to empower and support what makes life meaningful for them.

Participants mentioned several other characteristics that they use in practice to prevent cultural insensitivity and support a collaborative approach to healthcare. Table 1 lists these facilitating characteristics and quotes to explain the substance of their benefit.

Consensus among participants indicated that the use of these protective factors to promote cultural sensitivity and apply them in practice is not standardized. When asked, all but two participants said they had not received any culturally-based training since beginning their practice. Instead, they referred to developing skills through “on the job training” or “off the cuff learning.” The general way of medicine, one participant remarked, was to “throw you to the fire.” This suggested that use of standardized cultural humility training modules for healthcare providers was not common practice. Many attributed this to time constraints.

Individual efforts to gain culturally appropriate skills or enhance cultural humility were mentioned, however. For example, three participants reported that they attended medical conferences to discuss cultural challenges within medicine, one participant sought out cultural education within their organization, and another was invited by Native American community members to engage in traditional peace ceremonies. Participants described these additional efforts as uncommon and outside the parameters of a provider’s job responsibilities, as they require time commitments without compensation.

Additionally, eight participants said they share their personal contact information with patients so they may call them directly for medical needs. The conditions and frequency with which this is done was variable and more common among providers in specialized areas of medicine or those who described having a manageable patient panel. All who reported that they shared their personal contact information described it as an aspect of rural health service delivery that is atypical in other, non-rural healthcare systems.

Theme 3: Communication between healthcare providers is systematically fragmented

Healthcare is complex and multi-disciplinary, and patients’ treatment is rarely overseen by a single provider [ 82 ]. The array of provider types and specialties is vast, as is the range of responsibilities ascribed to providers. Thus, open communication among providers both within and between healthcare systems is vital for the success of collaborative healthcare [ 83 ]. Without effective communication achieved between healthcare providers, the appropriate delivery of healthcare services may be become compromised. Our participants noted that they face multiple challenges that complicate communication with other providers. Miscommunication between departments, often implicating the Emergency Department (ED), was a recurring point noted among participants. One participant who is a primary care physician said:

If one of my patients goes to the ER, I don’t always get the notes. They’re supposed to send them to the patient’s primary care doc. The same thing happens with general admissions, but again, I often find out from somebody else that my patient was admitted to the hospital.

This failure to communicate can negatively impact the patient, particularly if time sensitivity or medical complexity is essential to treatment. A patient’s primary care physician is the most accurate source of their medical history; without an effective way to obtain and synthesize a patient’s health information, there may be increased risk of medical error. One participant in a specialty field stated:

One of the biggest barriers I see is obtaining a concise description of a patient’s history and needs. You can imagine if you’re a mom and you’ve got a complicated kid. You head to the ER. The ER doc looks at you with really wide eyes, not knowing how to get information about your child that’s really important.

This concern was highlighted with a specific example from a different participant:

I have been unable to troubleshoot instances when I send people to the ER with a pretty clear indication for admission, and then they’re sent home. For instance, I had an older fellow with pretty severe chronic kidney disease. He presented to another practitioner in my office with shortness of breath and swelling and appeared to have newly onset decompensated heart failure. When I figured this out, I sent him to the ER, called and gave my report. The patient later came back for follow up to find out not only that they had not been admitted but they lost no weight with outpatient dialysis . I feel like a real opportunity was missed to try to optimize the care of the patient simply because there was poor communication between myself and the ER. This poor guy… He ended up going to the ER four times before he got admitted for COVID-19.

In some cases, communication breakdown was reported as the sole cause of a poor outcome. When communication is effective, each essential member of the healthcare team is engaged and collaborating with the same information. Some participants called this process ‘rounds’ when a regularly scheduled meeting is staged between a group of providers to ensure access to accurate patient information. Accurate communication may also help build trust and improve a patient’s experience. In contrast, ineffective communication can result in poor clarity regarding providers’ responsibilities or lost information. Appropriate delivery of healthcare considers the fit between providers and a patient’s specific healthcare needs; the factors noted here suggest that provider-provider miscommunication can adversely affect this dimension of healthcare access.

Another important mechanism of communication is the sharing of electronic medical records (EMRs), a process that continues to shift with technological advances. Innovation is still recent enough, however, for several of our study participants to be able to recall a time when paper charts were standard. Widespread adoption and embrace of the improvements inherent in electronic medical records expanded in the late 2000’s [ 84 ]. EMRs vastly improved the ability to retain, organize, safeguard, and transfer health information. Every participant highlighted EMRs at one point or another and often did so with an underlying sense of anger or frustration. Systematic issues and problems with EMRs were discussed. One participant provided historical context to such records:

Years back, the government aimed to buy an electronic medical record system, whichever was the best, and a number of companies created their own. Each were a reasonable system, so they all got their checks and now we have four completely separate operating systems that do not talk to each other. The idea was to make a router or some type of relay that can share information back and forth. There was no money in that though, so of course, no one did anything about it. Depending on what hospital, clinic or agency you work for, you will most likely work within one of these systems. It was a great idea; it just didn’t get finished.

Seven participants confirmed these points and their impacts on making coordination more difficult, relying on outdated communication strategies more often than not. Many noted this even occurs between facilities within the same city and in separate small metropolitan areas across the state. One participant said:

If my hospital decides to contract with one EMR and the hospital across town contracts with another, correspondence between these hospitals goes back to traditional faxing. As a provider, you’re just taking a ‘fingered crossed’ approach hoping that the fax worked, is picked up, was put in the appropriate inbox and was actually looked at. Information acquisition and making sure it’s timely are unforeseen between EMRs.

Participants reported an “astronomic” number of daily faxes and telephone calls to complete the communication EMRs were initially designed to handle. These challenges are even more burdensome if a patient moves from out of town or out of state; obtaining their medical records was repeatedly referred to as a “chore” so onerous that it often remains undone. Another recurring concern brought up by participants regarded accuracy within EMRs to lend a false sense of security. They are not frequently updated, not designed to be family-centered and not set up to do anything automatically. One participant highlighted these limitations by stating:

I was very proud of a change I made in our EMR system [EPIC], even though it was one I never should have had to make. I was getting very upset because I would find out from my nursing assistant who read the obituary that one of my patients had died. There was a real problem with the way the EMR was notifying PCP’s, so I got an EPIC-level automated notification built into our EMR so that any time a patient died, their status would be changed to deceased and a notification would be sent to their PCP. It’s just really awful to find out a week later that your patient died, especially when you know these people and their families really well. It’s not good care to have blind follow up.

Whether it be a physical or electronic miscommunication between healthcare providers, the appropriate delivery of healthcare can be called to question

Theme 4: Time and resource constraints disproportionately harm rural health systems

Several measures of system capacity suggest the healthcare system in the US is under-resourced. There are fewer physicians and hospital beds per capita compared to most comparable countries, and the growth of healthcare provider populations has stagnated over time [ 15 ]. Rural areas, in particular, are subject to resource limitations [ 16 ]. All participants discussed provider shortages in detail. They described how shortages impact time allocation in their day-to-day operations. Tasks like patient intakes, critical assessments, and recovering information from EMRs take time, of which most participants claimed to not have enough of. There was also a consensus in having inadequate time to spend on medically complex cases. Time pressures were reported to subsequently influence quality of care. One participant stated:

With the constant pace of medicine, time is not on your side. A provider cannot always participate in an enriching dialogue with their patients, so rather than listen and learn, we are often coerced into the mindset of ‘getting through’ this patient so we can move on. This echoes for patient education during discharge, making the whole process more arduous than it otherwise could be if time and resources were not as sparse.

Depending on provider type, specialty, and the size of patient panels, four participants said they have the luxury of extending patient visits to 40 + minutes. Any flexibility with patient visits was regarded as just that: a luxury. Very few providers described the ability to coordinate their schedules as such. This led some study participants to limit the number of patients they serve. One participant said:

We simply don’t have enough clinicians, which is a shame because these people are really skilled, exceptional, brilliant providers but are performing way below their capacity. Because of this, I have a smaller case load so I can engage in a level of care that I feel is in the best interest of my patients. Everything is a tradeoff. Time has to be sacrificed at one point or another. This compromise sets our system up to do ‘ok’ work, not great work.

Of course, managing an overly large number of patients with high complexity is challenging. Especially while enduring the burden of a persisting global pandemic, participants reflected that the general outlook of administering healthcare in the US is to “do more with less.” This often forces providers to delegate responsibilities, which participants noted has potential downsides. One participant described how delegating patient care can cause problems.

Very often will a patient schedule a follow up that needs to happen within a certain time frame, but I am unable to see them myself. So, they are then placed with one of my mid-level providers. However, if additional health issues are introduced, which often happens, there is a high-risk of bounce-back or need to return once again to the hospital. It’s an inefficient vetting process that falls to people who may not have specific training in the labs and imaging that are often included in follow up visits. Unfortunately, it’s a forlorn hope to have a primary care physician be able to attend all levels of a patient’s care.

Several participants described how time constraints stretch all healthcare staff thin and complicate patient care. This was particularly important among participants who reported having a patient panel exceeding 1000. There were some participants, however, who praised the relationships they have with their nurse practitioners and physician’s assistants and mark transparency as the most effective way to coordinate care. Collectively, these clinical relationships were built over long standing periods of time, a disadvantage to providers at the start of their medical career. All but one participant with over a decade of clinical experience mentioned the usefulness of these relationships. The factors discussed in Theme 4 are directly linked to the Availability dimension of access to healthcare. A patient’s ability to reach care is subject to the capacity of their healthcare provider(s). Additionally, further analysis suggests these factors also link to the Appropriateness dimension because the quality of patient-provider relationships may be negatively impacted if a provider’s time is compromised.

Theme 5: Profits are prioritized over addressing barriers to healthcare access in the US.

The US healthcare system functions partially for-profit in the public and private sectors. The federal government provides funding for national programs such as Medicare, but a majority of Americans access healthcare through private employer plans [ 85 ]. As a result, uninsurance rates influence healthcare access. Though the rate of the uninsured has dropped over the last decade through expansion of the Affordable Care Act, it remains above 8 percent [ 86 ]. Historically, there has been ethical criticism in the literature of a for-profit system as it is said to exacerbate healthcare disparities and constitute unfair competition against nonprofit institutions. Specifically, the US healthcare system treats healthcare as a commodity instead of a right, enables organizational controls that adversely affect patient-provider relationships, undermines medical education, and constitutes a medical-industrial complex that threatens influence on healthcare-related public policy [ 87 ]. Though unprompted by the interviewer, participants raised many of these concerns. One participant shared their views on how priorities stand in their practice:

A lot of the higher-ups in the healthcare system where I work see each patient visit as a number. It’s not that they don’t have the capacity to think beyond that, but that’s what their role is, making sure we’re profitable. That’s part of why our healthcare system in the US is as broken as it is. It’s accentuated focus on financially and capitalistically driven factors versus understanding all these other barriers to care.

Eight participants echoed a similar concept, that addressing barriers to healthcare access in their organizations is largely complicated because so much attention is directed on matters that have nothing to do with patients. A few other participants supported this by alluding to a “cherry-picking” process by which those at the top of the hierarchy devote their attention to the easiest tasks. One participant shared an experience where contrasting work demands between administrators and front-line clinical providers produces adverse effects:

We had a new administrator in our hospital. I had been really frustrated with the lack of cultural awareness and curiosity from our other leaders in the past, so I offered to meet and take them on a tour of the reservation. This was meant to introduce them to kids, families and Tribal leaders who live in the area and their interface with healthcare. They declined, which I thought was disappointing and eye-opening.

Analysis of these factors suggest that those who work directly with patients understand patient needs better than those who serve in management roles. This same participant went on to suggest an ulterior motive for a push towards telemedicine, as administrators primarily highlight the benefit of billing for virtual visits instead of the nature of the visits themselves.

This study explored barriers and facilitators to healthcare access from the perspective of rural healthcare providers in Montana. Our qualitative analysis uncovered five key themes: 1) a friction exists between aspects of patients’ rural identities and healthcare systems; 2) facilitating access to healthcare requires application of and respect for cultural differences; 3) communication between healthcare providers is systematically fragmented; 4) time and resource constraints disproportionately harm rural health systems; and 5) profits are prioritized over addressing barriers to healthcare access in the US. Themes 2 and 3 were directly supported by earlier qualitative studies that applied Levesque’s framework, specifically regarding healthcare providers’ poor interpersonal quality and lack of collaboration with other providers that are suspected to result from a lack of provider training [ 67 , 70 ]. This ties back to the importance of cultural humility, which many previous culture-based trainings have referred to as cultural competence. Cultural competence is achieved through a plethora of trainings designed to expose providers to different cultures’ beliefs and values but induces risk of stereotyping and stigmatizing a patient’s views. Therefore, cultural humility is the preferred idea, by which providers reflect and gain open-ended appreciation for a patient’s culture [ 88 ].

Implications for Practice

Perhaps the most substantial takeaway is how embedded rugged individualism is within rural patient populations and how difficult that makes the delivery of care in rural health systems. We heard from participants that stoicism and perceptions of stigma within the system contribute to this, but other resulting factors may be influential at the provider- and organizational-levels. Stoicism and perceived stigma both appear to arise, in part, from an understandable knowledge gap regarding the care system. For instance, healthcare providers understand the relations between primary and secondary care, but many patients may perceive both concepts as elements of a single healthcare system [ 89 ]. Any issue experienced by a patient when tasked to see both a primary and secondary provider may result in a patient becoming confused [ 90 ]. This may also overlap with our third theme, as a disjointed means of communication between healthcare providers can exacerbate patients’ negative experiences. One consideration to improve this is to incorporate telehealth programs into an existing referral framework to reduce unnecessary interfacility transfers; telehealth programs have proven effective in rural and remote settings [ 91 ].

In fact, telehealth has been rolled out in a variety of virtual platforms throughout its evolution, its innovation matched with continued technological advancement. Simply put, telehealth allows health service delivery from a distance; it allows knowledge and practice of clinical care to be in a different space than a patient. Because of this, a primary benefit of telehealth is its impact on improving patient-centered outcomes among those living in rural areas. For instance, text messaging technology improves early infant diagnosis, adherence to recommended diagnostic testing, and participant engagement in lifestyle change interventions [ 92 , 93 , 94 ]. More sophisticated interventions have found their way into smartphone-based technology, some of which are accessible even without an internet connection [ 95 , 96 ]. Internet accessibility is important because a number of study participants noted internet connectivity as a barrier for patients who live in low resource communities. Videoconferencing is another function of telehealth that has delivered a variety of health services, including those which are mental health-specific [ 97 ], and mobile health clinics have been used in rural, hard-to-reach settings to show the delivery of quality healthcare is both feasible and acceptable [ 98 , 99 , 100 ]. While telehealth has potential to reduce a number of healthcare access barriers, it may not always address the most pressing healthcare needs [ 101 ]. However, telehealth does serve as a viable, cost-effective alternative for rural populations with limited physical access to specialized services [ 102 ]. With time and resource limitations acknowledged as a key theme in our study, an emphasis on expanding telehealth services is encouraged as it will likely have significant involvement on advancing healthcare in the future, especially as the COVID-19 pandemic persists [ 103 ].

Implications for Policy

One could argue that most of the areas of fragmentation in the US healthcare system can be linked to the very philosophy on which it is based: an emphasis on profits as highest priority. Americans are, therefore, forced to navigate a health service system that does not work solely in their best interests. It is not surprising to observe lower rates of healthcare usage in rural areas, which may be a result from rural persons’ negative views of the US healthcare system or a perception that the system does not exist to support wellness. These perceptions may interact with ‘rugged individualism’ to squelch rural residents’ engagement in healthcare. Many of the providers we interviewed for this study appeared to understand this and strived to improve their patients’ experiences and outcomes. Though these efforts are admirable, they may not characterize all providers who serve in rural areas of the US. From a policy standpoint, it is important to recognize these expansive efforts from providers. If incentives were offered to encourage maximum efforts be made, it may lessen burden due to physician burnout and fatigue. Of course, there is no easy fix to the persisting limit of time and resources for providers, problems that require workforce expansion. Ultimately, though, the current structure of the US healthcare system is failing rural America and doing little to help the practice of rural healthcare providers.

Implications for Future Research

It is important for future health systems research efforts to consider issues that arise from both individual- and system-level access barriers and where the two intersect. Oftentimes, challenges that appear linked to a patient or provider may actually stem from an overarching system failure. If failures are critically and properly addressed, we may refine our understanding of what we can do in our professional spaces to improve care as practitioners, workforce developers, researchers and advocates. This qualitative study was exploratory in nature. It represents a step forward in knowledge generation regarding challenges in access to healthcare for rural Americans. Although mental health did not come up by design in this study, future efforts exploring barriers to healthcare access in rural systems should focus on access to mental healthcare. In many rural areas, Montana included, rates of suicide, substance use and other mental health disorders are highly prevalent. These characteristics should be part of the overall discussion of access to healthcare in rural areas. Optimally, barriers to healthcare access should continue to be explored through qualitative and mixed study designs to honor its multi-dimensional stature.

Strengths and Limitations

It is important to note first that this study interviewed healthcare providers instead of patients, which served as both a strength and limitation. Healthcare providers were able to draw on numerous patient-provider experiences, enabling an account of the aggregate which would have been impossible for a patient population. However, accounts of healthcare providers’ perceptions of barriers to healthcare access for their patients may differ from patients’ specific views. Future research should examine acceptability- and appropriateness-related barriers to healthcare access in patient populations. Second, study participants were recruited through convenience sampling methods, so results may be biased towards healthcare providers who are more invested in addressing barriers to healthcare access. Particularly, the providers interviewed for this study represented a subset who go beyond expectations of their job descriptions by engaging with their communities and spending additional uncompensated time with their patients. It is likely that a provider who exhibits these behavioral traits is more likely to participate in research aimed at addressing barriers to healthcare access. Third, the inability to conduct face-to-face interviews for our qualitative study may have posed an additional limitation. It is possible, for example, that in-person interviews might have resulted in increased rapport with study participants. Notwithstanding this possibility, the remote interview format was necessary to accommodate health risks to the ongoing COVID-19 pandemic. Ultimately, given our qualitative approach, results from our study cannot be generalizable to all rural providers’ views or other rural health systems. In addition, no causality can be inferred regarding the influence of aspects of rurality on access. The purpose of this exploratory qualitative study was to probe research questions for future efforts. We also acknowledge the authors’ roles in the research, also known as reflexivity. The first author was the only author who administered interviews and had no prior relationships with all but one study participant. Assumptions and pre-dispositions to interview content by the first author were regularly addressed throughout data analysis to maintain study integrity. This was achieved by conducting analysis by unique interview question, rather than by unique participant, and recoding the numerical order of participants for each question. Our commitment to rigorous qualitative methods was a strength for the study for multiple reasons. Conducting member checks with participants ensured trustworthiness of findings. Continuing data collection to data saturation ensured dependability of findings, which was achieved after 10 interviews and confirmed after 2 additional interviews. We further recognize the heterogeneity in our sample of participants, which helped generate variability in responses. To remain consistent with appropriate means of presenting results in qualitative research however, we shared minimal demographic information about our study participants to ensure confidentiality.

The divide between urban and rural health stretches beyond a disproportionate allocation of resources. Rural health systems serve a more complicated and hard-to-reach patient population. They lack sufficient numbers of providers to meet population health needs. These disparities impact collaboration between patients and providers as well as the delivery of acceptable and appropriate healthcare. The marker of rurality complicates the already cumbersome challenge of administering acceptable and appropriate healthcare and impediments stemming from rurality require continued monitoring to improve patient experiences and outcomes. Our qualitative study explored rural healthcare providers’ views on some of the social, cultural, and programmatic factors that influence access to healthcare among their patient populations. We identified five key themes: 1) a friction exists between aspects of patients’ rural identities and healthcare systems; 2) facilitating access to healthcare requires application of and respect for cultural differences; 3) communication between healthcare providers is systematically fragmented; 4) time and resource constraints disproportionately harm rural health systems; and 5) profits are prioritized over addressing barriers to healthcare access in the US. This study provides implications that may shift the landscape of a healthcare provider’s approach to delivering healthcare. Further exploration is required to understand the effects these characteristics have on measurable patient-centered outcomes in rural areas.

Availability of data and materials

The datasets generated and/or analyzed during the current study are not publicly available due to individual privacy could be compromised but are available from the corresponding author on reasonable request.

Ethics approval and consent to participate.

All study procedures and methods were carried out in accordance with relevant guidelines and regulations from the World Medical Association Declaration of Helsinki. Ethics approval was given by exempt review from the Institutional Review Board (IRB) at the University of Montana (IRB Protocol No.: 186–20). Participants received oral and written information about the study prior to interview, which allowed them to provide informed consent for the interviews to be recorded and used for qualitative research purposes. No ethical concerns were experienced in this study pertaining to human subjects.

Consent for publication.

The participants consented to the publication of de-identified material from the interviews.

Riley WJ. Health Disparities: Gaps in Access, Quality and Affordability of Medical Care. Trans Am Clin Climatol Assoc. 2012;123:167–74.

PubMed PubMed Central Google Scholar

Levesque J, Harris MF, Russell G. Patient-centred access to health care: conceptualising access at the interface of health systems and populations. Int J Equity Health. 2013;12:18. https://doi.org/10.1186/1475-9276-12-18 .

Article PubMed PubMed Central Google Scholar

Serban N. A Multidimensional Framework for Measuring Access. In: Serban N. Healthcare System Access: Measurement, Inference, and Intervention. New Jersey: John Wiley & Sons; 2019. p. 13–59.

Google Scholar

Roncarolo F, Boivin A, Denis JL, et al. What do we know about the needs and challenges of health systems? A scoping review of the international literature. BMC Health Serv Res. 2017;17:636. https://doi.org/10.1186/s12913-017-2585-5 .

Cabrera-Barona P, Blaschke T. Kienberger S. Explaining Accessibility and Satisfaction Related to Healthcare: A Mixed-Methods Approach. Soc Indic Res. 2017;133,719–739. https://doi.org/10.1007/s11205-016-1371-9 .

Coombs NC, Meriwether WE, Caringi J, Newcomer SR. Barriers to healthcare access among U.S. adults with mental health challenges: A population-based study. SSM Popul Health. 2021;15:100847. https://doi.org/10.1016/j.ssmph.2021.100847 .

Farietta TP, Lu B, Tumin R. Ohio’s Medicaid Expansion and Unmet Health Needs Among Low-Income Women of Reproductive Age. Matern Child Health. 2018;22(12):1771–9. https://doi.org/10.1007/s10995-018-2575-1 .

Article Google Scholar

Sommers BD, Blendon RJ, Orav EJ, Epstein AM. Changes in Utilization and Health Among Low-Income Adults After Medicaid Expansion or Expanded Private Insurance. JAMA Intern Med. 2016;1;176(10):1501–1509. https://doi.org/10.1001/jamainternmed.2016.4419 .

National Alliance on Mental Illness: The Doctor is Out. Continuing Disparities in Access to Mental and Physical Health Care. 2017. https://www.nami.org/Support-Education/Publications-Reports/Public-Policy-Reports/The-Doctor-is-Out . Accessed 29 Aug 2021.

Pollitz K, Lopes L, Kearney A, Rae M, Cox C, Fehr F, Rousseau D, Kaiser Family Foundation. US Statistics on Surprise Medical Billing. JAMA. 2020;323(6):498. https://doi.org/10.1001/jama.2020.0065 .

Kahraman C, Orobello C, Cirella GT. hanging Dynamics with COVID-19: Future Outlook. In: XX Cirella GT, editor. Human Settlements: Urbanization, Smart Sector Development, and Future Outlook. Singapore: Springer; 2022. p. 235–52.

Book Google Scholar

Blumenthal D, Fowler EJ, Abrams M, Collins SR. Covid-19 — Implications for the Health Care System. N Engl J Med. 2020;383(15):1483–8. https://doi.org/10.1056/NEJMsb2021088 .

Article CAS PubMed Google Scholar

Defêche J, Azarzar S, Mesdagh A, Dellot P, Tytgat A, Bureau F, Gillet L, Belhadj Y, Bontems S, Hayette M-P, Schils R, Rahmouni S, Ernst M, Moutschen M, Darcis G. In-Depth Longitudinal Comparison of Clinical Specimens to Detect SARS-CoV-2. Pathogens. 2021;10(11):1362. https://doi.org/10.3390/pathogens10111362 .

Article CAS PubMed PubMed Central Google Scholar

Rosenblatt RA, Hart LG. Physicians and rural America. West J Med. 2000;173(5):348–51. https://doi.org/10.1136/ewjm.173.5.348 .

Kaiser Family Foundation Analysis of OECD Data. 2020. https://www.healthsystemtracker.org/chart-collection/u-s-health-care-resources-compare-countries/#item-physicians-density-per-1000-population-2000-2018 . Accessed 31 Oct 2021.

Government Accounting Office. Physician Workforce: Locations and Types of Graduate Training Were Largely Unchanged, and Federal Efforts May Not Be Sufficient to Meet Needs. 2017. https://www.gao.gov/assets/gao-17-411.pdf . Accessed 31 Oct 2021.

The Kaiser Family Foundation: The Uninsured in Rural America. 2003. https://www.kff.org/wp-content/uploads/2013/01/the-uninsured-in-rural-america-update-pdf.pdf . Accessed 15 Sept 2021.

Wright B, Potter AJ, Trivedi AN, Mueller KJ. The Relationship Between Rural Health Clinic Use and Potentially Preventable Hospitalizations and Emergency Department Visits Among Medicare Beneficiaries. J Rural Health. 2018;34(4):423–30. https://doi.org/10.1111/jrh.12253 .

Article PubMed Google Scholar

Weichelt B, Bendixsen C, Patrick T. A Model for Assessing Necessary Conditions for Rural Health Care’s Mobile Health Readiness: Qualitative Assessment of Clinician-Perceived Barriers. JMIR Mhealth Uhealth. 2019;7(11): e11915. https://doi.org/10.2196/11915 .

Mangundu M, Roets L, Janse van Rensberg E. Accessibility of healthcare in rural Zimbabwe: The perspective of nurses and healthcare users. Afr J Prim Health Care Fam Med. 2020;12(1):e1-e7. https://doi.org/10.4102/phcfm.v12i1.2245 .

Rahayu YYS, Araki T, Rosleine D. Factors affecting the use of herbal medicines in the universal health coverage system in Indonesia. J Ethnopharmacol. 2020;260: 112974. https://doi.org/10.1016/j.jep.2020.112974 .

Bolin JN, Bellamy GR, Ferdinand AO, et al. Rural Healthy People 2020: New Decade, Same Challenges. Journal of Rural Health. 2015;31(3):326–33.

Reid S. The rural determinants of health: using critical realism as a theoretical framework. Rural Remote Health. 2019;19(3):5184. https://doi.org/10.22605/RRH5184 .

Wynia MK, Osborn CY. Health literacy and communication quality in health care organizations. J Health Commun. 2010;15 Suppl 2(Suppl 2):102–115. https://doi.org/10.1080/10810730.2010.499981 .

Centers for Disease Control and Prevention. What is Health Literacy? 2022. https://www.cdc.gov/healthliteracy/learn/index.html . Accessed 25 Feb 2022.

Davis TC, Arnold CL, Rademaker A, et al. Differences in barriers to mammography between rural and urban women. J Womens Health (Larchmt). 2012;21(7):748–55. https://doi.org/10.1089/jwh.2011.3397 .

Halverson J, Martinez-Donate A, Trentham-Dietz A, et al. Health literacy and urbanicity among cancer patients. J Rural Health. 2013;29(4):392–402. https://doi.org/10.1111/jrh.12018 .

Zahnd WE, Scaife SL, Francis ML. Health literacy skills in rural and urban populations. Am J Health Behav. 2009;33(5):550–7. https://doi.org/10.5993/ajhb.33.5.8 .

Dogba MJ, Dossa AR, Breton E, Gandonou-Migan R. Using information and communication technologies to involve patients and the public in health education in rural and remote areas: a scoping review. BMC Health Serv Res. 2019;19(1):128. https://doi.org/10.1186/s12913-019-3906-7 .

Committee on the Science of Changing Behavioral Health Social Norms, Board on Behavioral, Cognitive, and Sensory Sciences, Division of Behavioral and Social Sciences and Education; National Academies of Sciences, Engineering, and Medicine. In: Ending Discrimination Against People with Mental and Substance Use Disorders: The Evidence for Stigma Change. Washington: National Academics Press; 2016. p. 33–52.

Wu Y, Zhou H, Wang Q, Cao M, Medina A, Rozelle S. Use of maternal health services among women in the ethnic rural areas of western China. BMC Health Serv Res. 2019;19(1):179. https://doi.org/10.1186/s12913-019-3996-2 .

Meyer E, Hennink M, Rochat R, et al. Working Towards Safe Motherhood: Delays and Barriers to Prenatal Care for Women in Rural and Peri-Urban Areas of Georgia. Matern Child Health J. 2016;20(7):1358–65. https://doi.org/10.1007/s10995-016-1997-x .

Heinrich S. Medical science faces the post-truth era: a plea for the grassroot values of science. Curr Opin Anaesthesiol. 2020;33(2):198–202. https://doi.org/10.1097/ACO.0000000000000833 .

Esquivel MM, Chen JC, Woo RK, et al. Why do patients receive care from a short-term medical mission? Survey study from rural Guatemala. J Surg Res. 2017;215:160–6. https://doi.org/10.1016/j.jss.2017.03.056 .

Park LG, Dracup K, Whooley MA, et al. Symptom Diary Use and Improved Survival for Patients With Heart Failure [published correction appears in Circ Heart Fail. 2017 Dec;10(12):]. Circ Heart Fail. 2017;10(11):e003874. https://doi.org/10.1161/CIRCHEARTFAILURE.117.003874 .

Taleb F, Perkins J, Ali NA, et al. Transforming maternal and newborn health social norms and practices to increase utilization of health services in rural Bangladesh: a qualitative review. BMC Pregnancy Childbirth. 2015;15:75. https://doi.org/10.1186/s12884-015-0501-8 .

Billah SM, Hoque DE, Rahman M, et al. Feasibility of engaging “Village Doctors” in the Community-based Integrated Management of Childhood Illness (C-IMCI): experience from rural Bangladesh. J Glob Health. 2018;8(2): 020413. https://doi.org/10.7189/jogh.08.020413 .

Kaiser JL, Fong RM, Hamer DH, et al. How a woman’s interpersonal relationships can delay care-seeking and access during the maternity period in rural Zambia: An intersection of the Social Ecological Model with the Three Delays Framework. Soc Sci Med. 2019;220:312–21. https://doi.org/10.1016/j.socscimed.2018.11.011 .

Rao KD, Sheffel A. Quality of clinical care and bypassing of primary health centers in India. Soc Sci Med. 2018;207:80–8. https://doi.org/10.1016/j.socscimed.2018.04.040 .

Lee YT, Lee YH, Kaplan WA. Is Taiwan’s National Health Insurance a perfect system? Problems related to health care utilization of the aboriginal population in rural townships. Int J Health Plann Manage. 2019;34(1):e6–10. https://doi.org/10.1002/hpm.2653 .

Lyford M, Haigh MM, Baxi S, Cheetham S, Shahid S, Thompson SC. An Exploration of Underrepresentation of Aboriginal Cancer Patients Attending a Regional Radiotherapy Service in Western Australia. Int J Environ Res Public Health. 2018;15(2):337. https://doi.org/10.3390/ijerph15020337 .

Article PubMed Central Google Scholar

Rohr JM, Spears KL, Geske J, Khandalavala B, Lacey MJ. Utilization of Health Care Resources by the Amish of a Rural County in Nebraska. J Community Health. 2019;44(6):1090–7. https://doi.org/10.1007/s10900-019-00696-9 .

Johnston K, Harvey C, Matich P, et al. Increasing access to sexual health care for rural and regional young people: Similarities and differences in the views of young people and service providers. Aust J Rural Health. 2015;23(5):257–64. https://doi.org/10.1111/ajr.12186 .

Legido-Quigley H, Naheed A, de Silva HA, et al. Patients’ experiences on accessing health care services for management of hypertension in rural Bangladesh, Pakistan and Sri Lanka: A qualitative study. PLoS ONE. 2019;14(1): e0211100. https://doi.org/10.1371/journal.pone.0211100 .

Shaw B, Amouzou A, Miller NP, Bryce J, Surkan PJ. A qualitative exploration of care-seeking pathways for sick children in the rural Oromia region of Ethiopia. BMC Health Serv Res. 2017;17(1):184. https://doi.org/10.1186/s12913-017-2123-5 .

Larson E, Vail D, Mbaruku GM, Kimweri A, Freedman LP, Kruk ME. Moving Toward Patient-Centered Care in Africa: A Discrete Choice Experiment of Preferences for Delivery Care among 3,003 Tanzanian Women. PLoS ONE. 2015;10(8): e0135621. https://doi.org/10.1371/journal.pone.0135621 .

Spleen AM, Lengerich EJ, Camacho FT, Vanderpool RC. Health care avoidance among rural populations: results from a nationally representative survey. J Rural Health. 2014;30(1):79–88. https://doi.org/10.1111/jrh.12032 .

Weisgrau S. Issues in rural health: access, hospitals, and reform. Health Care Financ Rev. 1995;17(1):1–14.

CAS PubMed PubMed Central Google Scholar

National Center for Health Statistics. 2018. https://www.cdc.gov/nchs/surveys.htm . Accessed 29 Aug 2021.

Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res. 2017;17(1):88. https://doi.org/10.1186/s12913-017-2031-8 .

Dyer TA, Owens J, Robinson PG. The acceptability of healthcare: from satisfaction to trust. Community Dent Health. 2016;33(4):242–51. https://doi.org/10.1922/CDH_3902Dyer10 .

Padgett DK. Qualitative and Mixed Methods in Public Health. Thousand Oaks, California: SAGE Publications Inc.; 2012.

Tolley EE, Ulin PR, Mack N, Robinson ET, Succop SM. Qualitative Methods in Public Health: A Field Guide for Applied Research. 2nd ed. San Francisco, California: John Wiley & Sons, Inc.; 2016.

Ingram DD, Franco SJ. 2013 NCHS urban-rural classification scheme for counties. In: National Center for Health Statistics: Vital Health Statistics. 2014. https://www.cdc.gov/nchs/data/series/sr_02/sr02_166.pdf . Accessed 31 Oct 2021.

Health Resources & Services Administration. Health Professional Shortage Area Find. 2021. https://data.hrsa.gov/tools/shortage-area/hpsa-find . Accessed 31 Oct 2021.

Shaghaghi A, Bhopal RS, Sheikh A. Approaches to Recruiting “Hard-To-Reach” Populations into Re-search: A Review of the Literature. Health Promot Perspect. 2011;1(2):86–94. https://doi.org/10.5681/hpp.2011.009 .

Sadler GR, Lee HC, Lim RS, Fullerton J. Recruitment of hard-to-reach population subgroups via adaptations of the snowball sampling strategy. Nurs Health Sci. 2010;12(3):369–74. https://doi.org/10.1111/j.1442-2018.2010.00541.x .

Deterding NM, Waters MC. Flexible Coding of In-depth Interviews: A Twenty-first-century Approach. Sociological Methods & Research. 2021;50(2):708–39. https://doi.org/10.1177/0049124118799377 .

Schreier M, Stamann C, Janssen M, Dahl T, Whittal A. Qualitative Content Analysis: Conceptualizations and Challenges in Research Practice-Introduction to the FQS Special Issue" Qualitative Content Analysis I". InForum Qualitative Sozialforschung/Forum: Qualitative Social Research. 2019;20:26 DEU.

Guest G, Namey E, Chen M. A simple method to assess and report thematic saturation in qualitative research. PLoS ONE. 2020;15(5): e0232076. https://doi.org/10.1371/journal.pone.0232076 .

Mayring P. Qualitative Content Analysis. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research. 2000;1(2). https://doi.org/10.17169/fqs-1.2.1089 .

O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Academic Medicine, Vol. 89, No. 9 / Sept 2014. https://doi.org/10.1097/ACM.0000000000000388 .

Carter N, Bryant-Lukosius D, DiCenso A, Blythe J, Neville AJ. The use of triangulation in qualitative research. Oncol Nurs Forum. 2014;41(5):545–7. https://doi.org/10.1188/14.ONF.545-547 .

Miller S. Cultural humility is the first step to becoming global care providers. J Obstet Gynecol Neonatal Nurs. 2009;38(1):92–3. https://doi.org/10.1111/j.1552-6909.2008.00311.x .

Prasad SJ, Nair P, Gadhvi K, Barai I, Danish HS, Philip AB. Cultural humility: treating the patient, not the illness. Med Educ Online. 2016;21:30908. https://doi.org/10.3402/meo.v21.30908 .

George MS, Davey R, Mohanty I, Upton P. “Everything is provided free, but they are still hesitant to access healthcare services”: why does the indigenous community in Attapadi, Kerala continue to experience poor access to healthcare? Int J Equity Health. 2020;19(1):105. https://doi.org/10.1186/s12939-020-01216-1 .

Romanelli M, Hudson KD. Individual and systemic barriers to health care: Perspectives of lesbian, gay, bisexual, and transgender adults. Am J Orthopsychiatry. 2017;87(6):714–28. https://doi.org/10.1037/ort0000306 .

Bailie J, Schierhout G, Laycock A, et al. Determinants of access to chronic illness care: a mixed-methods evaluation of a national multifaceted chronic disease package for Indigenous Australians. BMJ Open. 2015;5(11): e008103. https://doi.org/10.1136/bmjopen-2015-008103 .

Reeve C, Humphreys J, Wakerman J, Carter M, Carroll V, Reeve D. Strengthening primary health care: achieving health gains in a remote region of Australia. Med J Aust. 2015;202(9):483–7. https://doi.org/10.5694/mja14.00894 .

Swan LET, Auerbach SL, Ely GE, Agbemenu K, Mencia J, Araf NR. Family Planning Practices in Appalachia: Focus Group Perspectives on Service Needs in the Context of Regional Substance Abuse. Int J Environ Res Public Health. 2020;17(4):1198. https://doi.org/10.3390/ijerph17041198 .

Kabia E, Mbau R, Oyando R, et al. “We are called the et cetera”: experiences of the poor with health financing reforms that target them in Kenya. Int J Equity Health. 2019;18(1):98. https://doi.org/10.1186/s12939-019-1006-2 .

Ho JW, Kuluski K, Im J. “It’s a fight to get anything you need” - Accessing care in the community from the perspectives of people with multimorbidity. Health Expect. 2017;20(6):1311–9. https://doi.org/10.1111/hex.12571 .

Latif A, Mandane B, Ali A, Ghumra S, Gulzar N. A Qualitative Exploration to Understand Access to Pharmacy Medication Reviews: Views from Marginalized Patient Groups. Pharmacy (Basel). 2020;8(2):73. https://doi.org/10.3390/pharmacy8020073 .

Tschirhart N, Diaz E, Ottersen T. Accessing public healthcare in Oslo, Norway: the experiences of Thai immigrant masseuses. BMC Health Serv Res. 2019;19(1):722. https://doi.org/10.1186/s12913-019-4560-9 .

Ireland S, Belton S, Doran F. “I didn’t feel judged”: exploring women’s access to telemedicine abortion in rural Australia. J Prim Health Care. 2020;12(1):49–56. https://doi.org/10.1071/HC19050 .

Stokes T, Tumilty E, Latu ATF, et al. Improving access to health care for people with severe chronic obstructive pulmonary disease (COPD) in Southern New Zealand: qualitative study of the views of health professional stakeholders and patients. BMJ Open. 2019;9(11): e033524. https://doi.org/10.1136/bmjopen-2019-033524 .

Jafar TH, Ramakrishnan C, John O, et al. Access to CKD Care in Rural Communities of India: a qualitative study exploring the barriers and potential facilitators. BMC Nephrol. 2020;21(1):26. https://doi.org/10.1186/s12882-020-1702-6 .

Doetsch J, Pilot E, Santana P, Krafft T. Potential barriers in healthcare access of the elderly population influenced by the economic crisis and the troika agreement: a qualitative case study in Lisbon, Portugal. Int J Equity Health. 2017;16(1):184. https://doi.org/10.1186/s12939-017-0679-7 .

Engel GL. The need for a new medical model: a challenge for biomedicine. Science. 1977;196(4286):129–36. https://doi.org/10.1126/science.847460 .

Risberg G, Hamberg K, Johansson E.E. Gender perspective in medicine: a vital part of medical scientific rationality. A useful model for comprehending structures and hierarchies within medical science. BMC Med. 2006;4(1):1.

Montana Census & Economic Information Center. 2021. https://ceic.mt.gov/ . Accessed 18 Sept 2021.

Smith M, Saunders R, Stuckhardt L, McGinnis JM, Committee on the Learning Health Care System in America, Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington: National Academies Press; 2013.

O’Daniel M, Rosenstein AH, Professional Communication and Team Collaboration. In: Hughes RG, editor. Chapter 33: Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Agency for Healthcare Research and Quality: Rockville; 2008.

Evans RS. Electronic Health Records: Then, Now, and in the Future. Yearb Med Inform. 2016;Suppl 1(Suppl 1):S48-S61. https://doi.org/10.15265/IYS-2016-s006 .

Tikkanen R, Osborn R, Mossialos E, Djordjevic A, Wharton GA. International Health Care System Profiles: United States. The Commonwealth Fund. 2020. https://www.commonwealthfund.org/international-health-policy-center/countries/united-states . Accessed 19 Sept 2021.

Berchick ER, Barnett JC, Upton RD. Health Insurance Coverage in the United States. Washington: United States Census Bureau; 2018.

Brock DW, Buchanan A. Ethical Issues in For-Profit Health Care. In: Gray BH, editor. For-Profit Enterprise in Health Care. Washington: National Academics Press; 1986.

Lekas HM, Pahl K, Fuller Lewis C. Rethinking Cultural Competence: Shifting to Cultural Humility. Health Serv Insights. 2020;13:1178632920970580. https://doi.org/10.1177/1178632920970580 .

Beaulieu MD. Primary and secondary care: Breaking down barriers for our patients with chronic diseases. Can Fam Physician. 2013;59(2):221.

Crafford L, Jenkins LS. Why seek a second consultation at an emergency centre? A qualitative study. Afr J Prim Health Care Fam Med. 2017;9(1):e1–8. https://doi.org/10.4102/phcfm.v9i1.1397 .

Sorensen MJ, von Recklinghausen FM, Fulton G, et al. Secondary overtriage: The burden of unnecessary interfacility transfers in a rural trauma system. JAMA Surg. 2013;148:763–8.

Sutcliffe CG, Thuma PE, van Dijk JH, et al. Use of mobile phones and text messaging to decrease the turnaround time for early infant HIV diagnosis and notification in rural Zambia: an observational study. BMC Pediatr. 2017;17(1):66. https://doi.org/10.1186/s12887-017-0822-z .

Baldwin LM, Morrison C, Griffin J, et al. Bidirectional Text Messaging to Improve Adherence to Recommended Lipid Testing. J Am Board Fam Med. 2017;30(5):608–14. https://doi.org/10.3122/jabfm.2017.05.170088 .

Albright K, Krantz MJ, Backlund Jarquín P, DeAlleaume L, Coronel-Mockler S, Estacio RO. Health Promotion Text Messaging Preferences and Acceptability Among the Medically Underserved. Health Promot Pract. 2015;16(4):523–32. https://doi.org/10.1177/1524839914566850 .

Cramer ME, Mollard EK, Ford AL, Kupzyk KA, Wilson FA. The feasibility and promise of mobile technology with community health worker reinforcement to reduce rural preterm birth. Public Health Nurs. 2018;35(6):508–16. https://doi.org/10.1111/phn.12543 .

Gbadamosi SO, Eze C, Olawepo JO, et al. A Patient-Held Smartcard With a Unique Identifier and an mHealth Platform to Improve the Availability of Prenatal Test Results in Rural Nigeria: Demonstration Study. J Med Internet Res. 2018;20(1): e18. https://doi.org/10.2196/jmir.8716 .

Trondsen MV, Tjora A, Broom A, Scambler G. The symbolic affordances of a video-mediated gaze in emergency psychiatry. Soc Sci Med. 2018;197:87–94. https://doi.org/10.1016/j.socscimed.2017.11.056 .

Schwitters A, Lederer P, Zilversmit L, et al. Barriers to health care in rural Mozambique: a rapid ethnographic assessment of planned mobile health clinics for ART. Glob Health Sci Pract. 2015;3(1):109–16. https://doi.org/10.9745/GHSP-D-14-00145 .

Kojima N, Krupp K, Ravi K, et al. Implementing and sustaining a mobile medical clinic for prenatal care and sexually transmitted infection prevention in rural Mysore, India. BMC Infect Dis. 2017;17(1):189. https://doi.org/10.1186/s12879-017-2282-3 .

Gorman SE, Martinez JM, Olson J. An assessment of HIV treatment outcomes among utilizers of semi-mobile clinics in rural Kenya. AIDS Care. 2015;27(5):665–8. https://doi.org/10.1080/09540121.2014.986053 .

Lee S, Black D, Held ML. Factors Associated with Telehealth Service Utilization among Rural Populations. J Health Care Poor Underserved. 2019;30(4):1259–72. https://doi.org/10.1353/hpu.2019.0104 .

Agha Z, Schapira RM, Maker AH. Cost effectiveness of telemedicine for the delivery of outpatient pulmonary care to a rural population. Telemed J E Health. 2002;8(3):281–91. https://doi.org/10.1089/15305620260353171 .

Doraiswamy S, Abraham A, Mamtani R, Cheema S. Use of Telehealth During the COVID-19 Pandemic: Scoping Review. J Med Internet Res. 2020;22(12): e24087. https://doi.org/10.2196/24087 .

Download references

Acknowledgements

This research was supported by the Center for Biomedical Research Excellence award (P20GM130418) from the National Institute of General Medical Sciences of the National Institute of Health. The first author was also supported by the University of Montana Burnham Population Health Fellowship. We would like to thank Dr. Christopher Dietrich, Dr. Jennifer Robohm and Dr. Eric Arzubi for their contributions on determining inclusion criteria for the healthcare provider population used for this study.

This research did not receive any specific grant from funding agencies in the public, commercial, and not-for-profit sectors.

Author information

Authors and affiliations.

School of Public & Community Health Sciences, University of Montana, 32 Campus Dr, Missoula, MT, 59812, USA

Nicholas C. Coombs & James Caringi

Department of Psychology, University of Montana, 32 Campus Dr, Missoula, MT, 59812, USA

Duncan G. Campbell

You can also search for this author in PubMed Google Scholar

Contributions

The authors confirm contribution to the paper as follows: study conception and design: NC and JC; data collection: NC; analysis and interpretation of results: NC and JC; draft manuscript preparation: NC, DC and JC; and manuscript editing: NC, DC and JC. All authors reviewed the results and approved the final version of the manuscript.

Corresponding author

Correspondence to Nicholas C. Coombs .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Coombs, N.C., Campbell, D.G. & Caringi, J. A qualitative study of rural healthcare providers’ views of social, cultural, and programmatic barriers to healthcare access. BMC Health Serv Res 22 , 438 (2022). https://doi.org/10.1186/s12913-022-07829-2

Download citation

Received : 11 January 2022

Accepted : 21 March 2022

Published : 02 April 2022

DOI : https://doi.org/10.1186/s12913-022-07829-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Access to healthcare
Rural health
Qualitative methods

BMC Health Services Research

ISSN: 1472-6963

General enquiries: [email protected]

Kristen Kennedy Terry and Bob Bayley Publish New Book Titled: "Social Network Analysis in Second Language Research Theory and Methods"

by Nicholas Bamshad Aoki
February 19, 2024

Congratulations to Dr. Kristen Kennedy Terry (UC Santa Cruz) and Dr. Bob Bayley (UC Davis) for publishing "Social Network Analysis in Second Language Research: Theory and Methods"! Click the link below for more information.

Link: https://www.routledge.com/Social-Network-Analysis-in-Second-Language-Research-Theory-and-Methods/KennedyTerry-Bayley/p/book/9781032005027

National Science Foundation - Where Discoveries Begin

Biological Sciences (BIO)
Computer and Information Science and Engineering (CISE)
Education and Human Resources (EHR)
Engineering (ENG)
Environmental Research and Education (ERE)
Geosciences (GEO)
Integrative Activities (OIA)
International Science and Engineering (OISE)
Mathematical and Physical Sciences (MPS)

Social, Behavioral and Economic Sciences (SBE)

Technology, Innovation and Partnerships (TIP)
Related Links
Interdisciplinary Research
NSF Organization List
Responsible and Ethical Conduct of Research
Staff Directory
Understanding NSF Research
About Funding
Archived Funding Search
Find Funding
Merit Review
Policies and Procedures
Preparing Proposals
Recent Opportunities
Transformative Research
Proposal and Award Policies and Procedures Guide (PAPPG)
Research.gov
Funding Opportunities For
Graduate Students
K-12 Educators
Postdoctoral Fellows
Undergraduate Students
Small Business
About Awards
Award Statistics (Budget Internet Info System)
Award Conditions
Managing Awards
Presidential and Honorary Awards
Search Awards
Public Access Initiative
All Documents
National Center for Science and Engineering Statistics (NCSES)
Obtaining Documents
Search Documents
For News Media
Multimedia Gallery
News Archive
Search News
Special Reports
Speeches and Lectures
About NSF Logo
Broadening Participation/Diversity
Budget and Performance
Career Opportunities
Contracting Opportunities
National Science Board (NSB)
NSF and Congress
NSF Toolkit
Office of Equity and Civil Rights
Organization List
Remote Participant Support
Transparency and Accountability
Social, Behavioral and Economic Sciences (SBE) Home
Behavioral and Cognitive Sciences (BCS)
About NCSES
Schedule of Release Dates
Corrections
Social and Economic Sciences (SES)
SBE Office of Multidisciplinary Activities (SMA)
Get NCSES Email Updates Get NCSES Email Updates Go
Contact NCSES
Research Areas
Share on X/Twitter
Share on Facebook
Share on LinkedIn
Send as Email

Replicating the Job Importance and Job Satisfaction Latent Class Analysis from the 2017 Survey of Doctorate Recipients with the 2015 and 2019 Cycles

Working papers are intended to report exploratory results of research and analysis undertaken by the National Center for Science and Engineering Statistics (NCSES). Any opinions, findings, conclusions, or recommendations expressed in this working paper do not necessarily reflect the views of the National Science Foundation (NSF). This working paper has been released to inform interested parties of ongoing research or activities and to encourage further discussion of the topic and is not considered to contain official government statistics.

This research was completed while Dr. Fritz was on academic leave from the University of Nebraska–Lincoln and participating in the NCSES Research Ambassador Program (formerly the Data Analysis and Statistics Research Program) administered by the Oak Ridge Institute for Science and Education (ORISE) and Oak Ridge Associate Universities (ORAU). Any opinions, findings, conclusions, or recommendations expressed in this working paper are solely the author’s and do not necessarily reflect the views of NCSES, NSF, ORISE, or ORAU.

Replication and reproducibility of results is a cornerstone of scientific research, as replication studies can identify artifacts that affect internal validity, investigate sampling error, increase generalizability, provide further testing of the original hypothesis, and evaluate claims of fraud. The purpose of the current working paper is to determine whether the five-class, three-response latent class solutions for job importance and job satisfaction found by Fritz (2022) using the 2017 cycle of the Survey of Doctorate Recipients data replicate using data from the 2015 and 2019 cycles. A series of latent class analyses were conducted using the Mplus statistical software, which determined that the five-class, three-response solutions for job importance and job satisfaction were also the best models for the 2015 and 2019 data. In addition, the class prevalences and response probabilities were highly consistent across time, indicating that the interpretation of the latent classes is the same across time as well. All of this gives strong evidence that the 2017 results were successfully replicated in the 2015 and 2019 data, increasing confidence in the original results and also setting the stage for a future longitudinal investigation of the latent classes using latent transition analysis.

Introduction

Background and rationale.

Replication and reproducibility are central tenets of science and scientific inquiry. Put simply, replication is the idea that if it is possible to carry out a research study more than once, then any person who exactly follows the same research protocol as the original study should (within a small margin of error) find the same results as the original study (Fidler and Wilcox 2018), whereas reproducibility is the idea that two people analyzing the same data using the same statistical methods should get the same results. While the concepts of scientific replication and reproducibility are simple, in practice, replication studies often completely or partially fail to replicate the results of previously published scientific studies, leading some to talk about a “replication crisis” in science (Fidler and Wilcox 2018; Pashler and Wagenmakers 2012) or state more strongly that, “It can be proven that most claimed research findings are false” (Ioannidis 2005). Fidler and Wilcox (2018) argue that this “replication crisis” is caused by five interrelated characteristics of the current scientific publication process: (1) the rarity of published replication studies in many fields, (2) the inability to reproduce the statistically significant results of many published studies, (3) a bias towards only publishing scientific studies that report statistically significant effects, (4) a lack of transparency and completeness with regard to sampling and analyses in published studies, and (5) the use of “questionable research practices” such as p hacking in order to obtain significant results. In the U.S. federal statistical system, issues with transparency and reproducibility are important enough that the NCSES tasked the Committee on National Statistics, part of the National Academies of Sciences, Engineering, and Medicine, to produce a consensus study report on the topic (NASEM 2021).

Regardless of the reason, many replication studies fail to reproduce the results of the original study; a consequence of increased focus on replicability is an increased emphasis on the need to reproduce the results from any individual study through the use of one or more replication studies, especially in the social sciences. In general, replication studies fall into two categories: exact replications, which seek to exactly replicate a prior study, and conceptual replications, which seek to determine whether the results of the original study generalize to a new population or context. Regardless of whether a replication study is exact or conceptual, Schmidt (2009) lists five functions of replications studies: (1) control for fraud, (2) control for sampling error, (3) control for artifacts that affect internal validity, (4) increase in generalizability, and (5) further testing of the original hypothesis. In the context of longitudinal survey work, the replication of results at multiple time points increases confidence in the original results and decreases the likelihood that the results at any individual point in time are due solely to artifacts specific to that time point or due to sampling error and measurement error. This increase in confidence, in turn, increases the generalizability of the original results. Ignoring for the moment cases of explicit fraud or data entry and analytic errors, failure to replicate the findings from data collected at one point in time at another point in time could be an indicator of time-specific artifacts or that the effect of interest is changing over time, both of which would require a deeper investigation of the longitudinal structure of the data as a whole.

The purpose of the current working paper is twofold. First is to determine whether the results from Fritz (2022), who found five latent job importance classes and five latent job satisfaction classes using the 2017 cycle of the Survey of Doctorate Recipients (SDR), replicate with the 2015 and 2019 cycle data. Specifically, this paper seeks to rule out the possibility that the results from Fritz (2022) were caused solely by time-specific artifacts (and to a lesser extent, sampling and measurement error) that affected only the 2017 data collection in order to increase confidence in and generalizability of the results of the original study. Second is to lay the groundwork for a future working paper investigating whether individuals move between latent job importance and job satisfaction classes across time (and if so, who moves between classes and in what direction) using latent transition analysis (LTA). Because the first step in conducting an LTA is to determine the latent class structure at each time point, this replication study also fulfills this requirement.

Participants

Survey questions, analyses and software.

The current working paper uses the publicly available microdata from the 2015, 2017, and 2019 cycles of the SDR; information about inclusion criteria and sampling are provided in the supporting documentation for the public use files (NCSES 2018, 2019, 2021). To participate in a specific SDR cycle, individuals must have completed a research doctorate in a science, engineering, or health (SEH) field from a U.S. academic institution prior to 1 July two calendar years previous to the current cycle year (e.g., prior to 1 July 2017 for the 2019 cycle). Participants must also be less than 76 years of age and not institutionalized or terminally ill on 1 February of the cycle year. All individuals who were selected to participate in a specific cycle and continued to meet the inclusion criteria were eligible to participate in later cycles, with each cycle’s sample supplemented by individuals who had graduated since the previous cycle. For the 2019 cycle, individuals who did not respond to either the 2015 or 2017 cycles were removed, while 14,564 individuals eligible but not selected for the 2015 cycle were added, and a new stratification design that strengthened reporting for minority groups and small SEH degree fields was utilized. For the final data set of each cycle, missing values were imputed using both logical and statistical methods, and sampling weights were calculated to adjust for the stratified sampling schema; unknown eligibility; nonresponse; and demographic characteristics including gender, race and ethnicity, location, degree year, and degree field.

Only participants who were asked to respond to the job importance and job satisfaction questions for each cycle were included in the current paper, resulting in final sample sizes of 78,286 individuals for the 2015 cycle, 85,720 individuals for the 2017 cycle, and 80,869 individuals for the 2019 cycle. Table 1 contains the sampling weight–estimated population frequencies by year for gender, physical disability, race and ethnicity, age (in 10-year increments), degree area, whether the participant was living in the United States at the time of data collection, workforce status, and employment sector. Note that these values are based on the reduced samples used for the current analyses and should not be considered official statistics. Although there are some differences across time, in general, the population remained relatively stable in regard to demographics, with the majority of SEH doctoral degree holders for all three cycles identifying as male, White, employed, holding a degree in science, and reporting no physical disability.

Population estimates of Survey of Doctorate Recipients participant characteristics, by selected years: 2015, 2017, and 2019

The values presented here are provided for reference only as they are based on applying the sampling weights to the reduced samples for each collection cycle used for the current project (2015: n = 78,286; 2017: n = 85,720; 2019: n = 80,869) and therefore do not match the official values reported by the National Center for Science and Engineering Statistics. All participants who identified as Hispanic were included in the Hispanic category, and only in the Hispanic category, regardless of whether they also identified with one or more of the racial categories. The Other category included individuals who identified as multiracial.

National Center for Science and Engineering Statistics, Survey of Doctorate Recipients, 2015, 2017, and 2019.

The current working paper focuses on two SDR questions: “When thinking about a job, how important is each of the following factors to you?” and “Thinking about your principal job held during the week of February 1, please rate your satisfaction with that job’s….” For each question, the participants were asked to rate nine job factors on a 4-point response scale. As in the final models for Working Paper NCSES 22-207 (Fritz 2022), the “somewhat unimportant” and “not important at all” options were combined into a single “unimportant” category and the “somewhat dissatisfied” and “very dissatisfied” options were combined into a single “dissatisfied” category resulting in three response categories for each question. Table 2 shows the sampling weight–estimated population response rates for the three response options for each of the nine job factors for importance and satisfaction. As with Table 1 , these values are based on the reduced samples used for the current analyses and should not be considered official statistics. Despite some small variability across time, in general, the response rates for each response option for each job factor were remarkably similar for all three cycles.

Population estimates of response proportions in percentages for importance of and satisfaction with nine job-related factors, by selected years: 2015, 2017, and 2019

Latent Class Analyses

The current paper uses latent class analysis (LCA), which has the mathematical model (Collins and Lanza 2010)

The probability of observing a specific response pattern on a set of indicators is equal to the product of the response probabilities, taken across all possible responses for all indicators for a specific latent class, multiplied by the prevalence for that class, and then summed across all of the latent classes.

where J is the number of indicators (i.e., job factors, so J = 9), R j is the number of response options for indicator j (here, R j = 3), C is the number of latent classes, γ c is the prevalence of class c , and ρ j,r j | c is the probability of giving response r j to indicator j for a member of class c . While previous work (Fritz 2022) has indicated that there are five job importance latent classes and five job satisfaction latent classes, given that the purpose of the current analyses is to test the veracity of this prior work, multiple models with differing numbers of latent classes were estimated and compared, and the correct number of classes to retain was based on four criteria: (1) percentage decrease in adjusted Bayesian Information Criterion (aBIC) value when an extra latent class was added, (2) solution stability across 1,000 random starts, (3) model entropy, and (4) interpretability of the latent classes.

Sampling weight–estimated population frequencies were computed using PROC SURVEYFREQ with the WEIGHT option in SAS 9.4 (SAS 2021). All LCA models were estimated with Mplus 8.4 (Muthèn and Muthèn 2021) using the maximum likelihood with robust standard errors estimator, treating the indicators as ordered categories (CATEGORICAL) and using 1,000 random starts, each of which was carried through all three estimation stages (STARTS ARE 1000 1000 1000;). Note that while Mplus and PROC LCA (Lanza et al. 2015) give almost identical results (e.g., the results from the 2017 five-class, three-response job importance LCA model in Mplus were identical to the same model in PROC LCA to at least three decimal places), Mplus calculates the aBIC based on the loglikelihood whereas PROC LCA calculates the aBIC based on the G 2 statistic, which changes the scaling of the aBIC values. As such, all LCA results for the 2017 SDR cycle reported here have been rerun in Mplus to put the 2017 model aBIC values on the same scale as the 2015 and 2019 model values.

Latent Classes: Job Importance

Latent classes: job satisfaction.

All LCA models with between one and eight latent classes were fit to the 9 three-response job importance factors separately for the 2015, 2017, and 2019 cycles of SDR data. All models converged normally, although not all 1,000 random starting values converged for the seven- and eight-class models. Table 3 contains the aBIC, percentage decrease in aBIC between models with C + 1 and C classes, entropy, and stability values for each model by year. As described previously, the aBIC values for the 2017 cycle do not match the values from Fritz (2022) because Mplus computes the aBIC values using the loglikelihood rather than G 2 . Table 3 reveals high consistency in model fit across years. These values indicate that the five-class solution fits the best for all three cycles. For example, adding an additional class reduces the aBIC value by 1.2% or more up through five classes, but adding a sixth latent class only reduces the aBIC value by 0.5% or less for each cycle. In addition, the stability for the five-class solution is highest of the four- through eight-class solutions for all three cycles, and the stability drops substantially for models with more than five classes while the entropy values stay approximately the same.

Model fit indices for job importance latent class analysis models, by selected years: 2015, 2017, and 2019

aBIC = adjusted Bayesian Information Criterion.

Stability is based on 1,000 random starts unless denoted with an asterisk (*), which indicates that not all 1,000 starts converged. When one or more starts failed to converge, stability is based on the number of starts out of 1,000 that did converge. The preferred 5-class solution is shown in bold.

The next step is to investigate the interpretation of the five-class solution for the 2015 and 2019 cycles as it is possible that there are five latent classes for each cycle, but that the interpretation of one or more classes is different in the 2017 cycle than the other cycles. The prevalence (i.e., estimated percentage of the population) for each class, as well as the response probabilities for each job factor, of the five-class, three response LCA model are shown in Table 4 by year. As with the previous tables, while the prevalances and response probabilities are not identical for each cycle, Table 4 shows a high level of consistency across years. The largest class for all three cycles is the Everything I s Very Important class whose members have a high probability of rating all nine job factors as “very important.” The second largest class for all three cycles is the Challenge and Independence A re More Important T han Salary and Benefits class whose members are most likely to rate a job’s intellectual challenge, level of independence, and contribution to society as “very important” and the job’s salary and benefits as only “somewhat important.” The Benefits and Salary A re More Important T han Responsibility class is the third largest class for all three cycles, and its members are mostly likely to rate a job’s salary, benefits, and security as “very important” and the job’s level of responsibility as “somewhat important.” The fourth largest class for all three cycles is the Everything I s Somewhat Important class whose members are most likely to rate all nine job factors as “somewhat important.” And the smallest class for each cycle is the Advancement, Security, and Benefits A re Unimportant class whose members are mostly likely to rate a job’s benefits, security, and opportunity for advancement as “unimportant,” although members of this group are likely to rate the job’s location as “very important.” Note that this smallest class is always larger than the 5% rule of thumb for retaining a class (Nasserinejad et al. 2017). Based on this, the five-class, three-response job importance solution reported by Fritz (2022) for the 2017 SDR data does replicate with the 2015 and 2019 cycles of the SDR.

Five-class job importance latent class analysis solution with three response options, by selected years: 2015, 2017, and 2019

Probabilities may not sum to 1.000 due to rounding. Response probabilities greater than or equal to 0.500 are considered salient and are represented in bold. Response probabilities more than twice as large as the next largest probability for that item for a specific year are highlighted in blue.

All LCA models with between one and eight latent classes were fit to the 9 three-response job satisfaction factors separately for the 2015, 2017, and 2019 cycles of SDR data. All models converged normally; again, a small percentage of the 1,000 random starting values did not converge, although just for the eight-class model. Table 5 shows the aBIC, percentage decrease in aBIC between models with C + 1 and C classes, entropy, and stability values for each model by year. As with the model fit indices for job importance, there is a high level of consistency in model fit across cycles for job satisfaction, and stability was 72.2% or higher for the one- through six-class solutions. Unlike the job importance models, however, adding a fifth class reduced the aBIC value by less than 1% for the job satisfaction models. It is important to remember that the decision to include the fifth job satisfaction class in Fritz (2022) was based more on the increased interpretability of the five-class solution compared to the four- and six-class solutions than the improvement in model fit. That is, while inclusion of the fifth class increased model fit modestly, the fifth class improved the separation of the other four classes, making them easier to interpret.

Model fit indices for job satisfaction latent class analysis models, by selected years: 2015, 2017, and 2019

Investigating the response probabilities and prevalences of the five-class model, shown in Table 6 , reveals that the interpretation of the five-class solution is identical for the 2015, 2017, and 2019 cycles, although the rank order of the smaller classes does vary (note that the order of classes in Table 6 is based on the 2017 prevalences). The largest class for all three cycles is the Very Satisfied W ith Independence, Challenge, and Responsibility class whose members are most likely to rate their satisfaction with their job’s level of independence, intellectual challenge, level of responsibility, and contribution to society as high but are less satisfied with their salary, benefits, and opportunities for advancement. The second largest class for all three cycles is the Very Satisfied W ith Everything class whose members report being very satisfied with all facets of their current job. While the three smaller classes vary in terms of rank, most likely due to sampling error as the prevalences for these three classes are very similar, the classes themselves are the same across time. Members of the Very Satisfied W ith Benefits class are most likely to rate their satisfaction with their job’s benefits, salary, and security as “very satisfied,” but only rate their satisfaction with their opportunity for advancement and level of responsibility as “somewhat satisfied.” Members of the Dissatisfied W ith Opportunities F or Advancement class are defined by their high probability of being dissatisfied with their opportunities for advancement in their current job. And the Somewhat Satisfied W ith Everything class members are mostly likely to rate their satisfaction with all of their job’s facets as “somewhat satisfied.” Based on these results, the five-class, three-response solution was determined to be the correct model for the 2015 and 2019 data, and, as a result, the five-class, three-response job satisfaction solution reported by Fritz (2022) for the 2017 SDR data does replicate with the 2015 and 2019 cycles of the SDR.

Five-class job satisfaction latent class analysis solution with three response options, by selected years: 2015, 2017, and 2019

There are three major take-aways from the results presented here. First, the replication was successful. As shown in Table 3 and Table 5 , the fit of the various LCA models is very similar across time, and Table 4 and Table 6 show that the prevalences and response probabilities (and hence, the interpretation) of the latent classes in the 2017 five-class solutions are almost identical to those for the 2015 and 2019 cycles for both job importance and job satisfaction. Perhaps this is unsurprising given the very similar response rates for each cycle shown in Table 2 , but all of this indicates that the five-class, three-response LCA solutions for the 2017 SDR data reported by Fritz (2022) do replicate for both job importance and job satisfaction in the 2015 and 2019 cycles of the SDR. While the replication of these five-class solutions provides evidence that these models were not selected solely due to an artifact of the 2017 SDR data, it is important to note that replicating the 2017 results with the 2015 and 2019 data does not rule out all alternative explanations. For example, because the SDR employs a longitudinal, repeated-measures design, with many of the doctoral degree holders who participated in the 2015 cycle also participating in the 2017 and 2019 cycles, it is possible the replication results are due to sampling error, and a different solution would be found with a different sample of doctoral degree holders. In addition, the replication says nothing about whether these five importance and five satisfaction classes are absolute in the sense that what doctoral degree holders in the latter half of the 2010s view as important, and their satisfaction with those job factors may not be the same as for doctoral degree holders in the 1980s or the 2040s. Regardless, replicating the 2017 results strengthens the validity and the generalizability of the original findings (Schmidt 2009).

Second, the replication highlights several findings from Fritz (2022) that, while reported in the original paper, were not as apparent until the results from the 2015, 2017, and 2019 cycles were considered together. For example, for all three cycles, over 70% of respondents were mostly likely to rate opportunities for advancement as “somewhat important” or “unimportant” ( Table 4 , Classes 2, 3, 4, and 5), but less than 30% of respondents were most likely to rate their satisfaction with their opportunities for advancement at their current job as “very satisfied” ( Table 6 , Class 2). This would indicate that most respondents were likely to believe their opportunities for advancement at their current job could be improved, and an important area of future research could be investigating why most doctoral degree holders feel this way and what would need to change in order for them to be very satisfied with their opportunities for advancement at their current job.

Another result that stands out in the replication concerns the job location. Table 4 shows that for all three cycles, over 85% of respondents were most likely to rate a job’s location as “very important” (Classes 1, 2, 3, and 5—only members of Class 4 are mostly likely to rate location as “somewhat important”), indicating that job location does not follow the intrinsic or extrinsic divide found by Fritz (2022) for Classes 2 and 3. The same pattern is seen in Table 6 for job satisfaction with members of Class 1, who are mostly likely to rate their satisfaction with only their job’s intrinsic facets as very high, and members of Class 3, who are mostly likely to rate their satisfaction with only their job’s extrinsic facets as very high, both being most likely to rate their satisfaction with their current job’s location as “very satisfied.” This would suggest that a job’s location is neither intrinsic nor extrinsic (or is somehow both). It is also possible that different respondents are interpretating the term “location” differently, with some conceptualizing location as country or state and others conceptualizing job location as a specific neighborhood or distance from their residence (e.g., length of daily commute). As with advancement, future research on doctoral degree holders could seek to better understand how job location relates to satisfaction.

Third, and finally, since the replication involved additional time points using the same sample, some potential longitudinal effects can be examined. For example, the size of some classes appears to systematically change over time, especially for job satisfaction. Most notably, the prevalence of the Satisfied W ith Independence, Challenge, and Responsibility (Class 1), Dissatisfied W ith Opportunities F or Advancement (Class 4), and Somewhat Satisfied W ith Everything (Class 5) classes all decrease from 2015 to 2019, whereas the size of the Satisfied W ith Everything (Class 2) and Satisfied W ith Benefits (Class 3) classes increase with each additional SDR cycle. While it is tempting to interpret this apparent trend to mean that overall job satisfaction is increasing or that average salary and benefits for doctoral degree holders is increasing, interpreting longitudinal effects in this replication study is problematic for several reasons. The most important issue is that the LCA models presented here do not model or statistically test any longitudinal hypotheses. As noted by Kenny and Zautra (1995), any individual’s score at a specific point in time in repeated-measures data is made up of three types of variability: trait, state, and error. Traits remain stable or change systematically across time; states are nonrandom, time-specific deviations from a trait; and errors are random deviations from a trait. For example, someone who has been in the same job for 10 years might be generally very satisfied with their current job for all 10 years (trait) but was dissatisfied on the day they filled out the SDR because they had an argument with a coworker the previous day (state) and selected “somewhat dissatisfied” because there was no “neither satisfied nor dissatisfied” response option (error).

Longitudinal models that can distinguish between trait, state, and error variability are therefore necessary to test and make conclusions about longitudinal effects. In more exact terms, while the replication of the 2017 solutions with the 2015 and 2019 data indicates that job importance and job satisfaction exhibit a high level of equilibrium that resulted in the same LCA solution at each time point, the replication does not provide any evidence for stability or stationarity of job satisfaction or importance across time. In this context, equilibrium is the consistency of the patterns of covariances and variances between items at a single point in time across repeated measurements (Dwyer 1983), while stability is the consistency of the mean level of a variable across time, and stationarity refers to an unchanging causal relationship between variables across time (Kenny 1979). Investigating whether the average level of satisfaction changed across time or whether the way in which satisfaction changed across time (i.e., the trajectory) differed by latent class would require the use of a latent growth curve model or a growth mixture model, respectively. In addition, the replication does not provide any insight into whether individuals tend to stay in the same class across time or whether individuals change classes, which would require the use of a latent transition model. Based on this, there are numerous longitudinal hypotheses that could be tested in order to gain a more complete understanding of the latent job importance and job satisfaction classes for doctoral degree holders.

Collins LM, Lanza ST. 2010. Latent Class and Latent Transition Analysis for Applications in the Social, Behavioral, and Health Sciences . Hoboken, NJ: Wiley.

Dwyer JH. 1983. Statistical Models for the Social and Behavioral Sciences . New York: Oxford University Press.

Fidler F, Wilcox J. 2018. Reproducibility in Scientific Results. In Zalta EN, editor, The Stanford Encyclopedia of Philosophy. Stanford, CA: Metaphysics Research Lab, Stanford University. Available at https://plato.stanford.edu/entries/scientific-reproducibility/#MetaScieEstaMoniEvalReprCris .

Fritz MS; National Center for Science and Engineering Statistics (NCSES). Job Satisfaction versus Job Importance: A Latent Class Analysis of the 2017 Survey of Doctorate Recipients. Working Paper NCSES 22-207. Arlington, VA: National Science Foundation. Available at https://www.nsf.gov/statistics/2022/ncses22207/ .

Ioannidis JPA. 2005. Why Most Research Findings Are False. PLoS Medicine 2(8):e124. Available at https://doi.org/10.1371/journal.pmed.0020124 .

Kenny DA. 1979. Correlation and Causality . New York: John Wiley & Sons.

Kenny DA, Zautra A. 1995. The Trait-State-Error Model for Multiwave Data. Journal of Consulting and Clinical Psychology 63(1):52–59. Available at https://doi.org/10.1037//0022-006x.63.1.52 .

Lanza ST, Dziak JJ, Huang L, Wagner A, Collins LM. 2015. PROC LCA & PROC LTA (Version 1.3.2) . [Software]. University Park, PA: Methodology Center, Pennsylvania State University. Available at https://www.latentclassanalysis.com/software/proc-lca-proc-lta/ .

Muthèn BO, Muthèn LK. 2021. Mplus (Version 8 .4) . [Software]. Los Angeles, CA: Muthèn & Muthèn.

Nasserinejad K, van Rosmalen J, de Kort W, Lesaffre E. 2017. Comparison of Criteria for Choosing the Number of Latent Classes in Bayesian Finite Mixture Models. PLoS ONE 12:1–23. Available at https://doi.org/10.1371/journal.pone.0168838 .

National Academies of Science, Engineering, and Medicine (NASEM). 2021. Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies . Washington, DC: National Academies Press. Available at https://www.nationalacademies.org/our-work/transparency-and-reproducibility-of-federal-statistics-for-the-national-center-for-science-and-engineering-statistics .

National Center for Science and Engineering Statistics (NCSES). 2018. Survey of Doctorate Recipients , 2015 . Data Tables. Alexandria, VA: National Science Foundation. Available at https://ncsesdata.nsf.gov/doctoratework/2015/ .

National Center for Science and Engineering Statistics (NCSES). 2019. Survey of Doctorate Recipients , 2017 . Data Tables. Alexandria, VA: National Science Foundation. Available at https://ncsesdata.nsf.gov/doctoratework/2017/ .

National Center for Science and Engineering Statistics (NCSES). 2021. Survey of Doctorate Recipients, 201 9 . NSF 21-320. Alexandria, VA: National Science Foundation. Available at https://ncses.nsf.gov/pubs/nsf21320/ .

Pashler H, Wagenmakers E-J. 2012. Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? Perspectives on Psychological Science 7(6):528–30. Available at https://doi.org/10.1177/1745691612465253 .

SAS. 2021. SAS (Version 9.4) . Cary, NC: SAS Institute Inc.

Schmidt S. 2009. Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences. Review of General Psychology 13(2):90–100. Available at https://doi.org/10.1037/a0015108 .

Suggested Citation

Fritz MS; National Center for Science and Engineering Statistics (NCSES). 2022. Replicating the Job Importance and Job Satisfaction Latent Class Analysis from the 2017 Survey of Doctorate Recipients with the 2015 and 2019 Cycles . Working Paper NCSES 22-208. Alexandria, VA: National Science Foundation. Available at https://ncses.nsf.gov/pubs/ncses22208 .

Matthew S. Fritz NCSES Research Ambassador/Fellow–Established Scientist, NCSES Associate Professor of Practice, Department of Educational Psychology, University of Nebraska–Lincoln E-mail: [email protected] or [email protected]

National Center for Science and Engineering Statistics Directorate for Social, Behavioral and Economic Sciences National Science Foundation 2415 Eisenhower Avenue, Suite W14200 Alexandria, VA 22314 Tel: (703) 292-8780 FIRS: (800) 877-8339 TDD: (800) 281-8749 E-mail: [email protected]

Read more about the source: Survey of Doctorate Recipients (SDR) .

Can't Find What You Are Looking For?

Search NCSES to find data and reports.

IMAGES

Social research and analysis
Definition and Types of Social Research Methods
Sociology GCSE Research Methods (9-1)
Social Research
Social Research Methods by H. Russell Bernard, Hardcover, 9781412978545
295 Sociology Research Topics and Tips to Consider

VIDEO

What is Social Research? A short video produced by TheSocRes
PMS: Social Work Lecture-10 Paper-2 ll Social Research
Why Social Work is a Scientific Study #socialwork #socialscience #research
Statistical Methods for Social Sciences l Statistical Methods and data analysis l @bismillahmujahid
Research Methods for Social Work
Importance of theory & Selecting a suitable framework for a study-1

COMMENTS

2.2 Research Methods
Recall the 6 Steps of the Scientific Method. Differentiate between four kinds of research methods: surveys, field research, experiments, and secondary data analysis. Explain the appropriateness of specific research approaches for specific topics. Sociologists examine the social world, see a problem or interesting pattern, and set out to study it.
Analytical Methods for Social Research
About Analytical Methods for Social Research. Analytical Methods for Social Research presents texts on empirical and formal methods for the social sciences. Volumes in the series address both the theoretical underpinnings of analytical techniques, as well as their application in social research. Some series volumes are broad in scope, cutting ...
Principles and Methods of Social Research
Through a multi-methodology approach, Principles and Methods of Social Research, Fourth Edition covers the latest research techniques and designs and guides readers toward the design and conduct of social research from the ground up.Applauded for its comprehensive coverage, the breadth and depth of content of this new edition is unparalleled.
Social Research Methods
This introduction to research methods provides students and researchers with unrivalled coverage of both quantitative and qualitative methods, making it invaluable for anyone embarking on social research. Bridging the gap between theory and practice, Social Research Methods, Fifth Edition, is packed full of engaging examples and practical tips to equip students with the tools and knowledge ...
Social Research Methods
Preview. This text provides a comprehensive guide to doing research in the social and behavioral sciences—from research design and sampling to collecting and analyzing data. Rich in examples, the book has been revised and updated to provide today's students with a conceptual understanding of each qualitative and quantitative technique, as ...
Social Research Methods
This is a comprehensive text for the research methods course taught out of sociology or other social science disciplines. It covers a wide range of methods and approaches to study design, data collection, and analysis. It is clear, comprehensive, and pays equal attention to both qualitative and quantitative methods, while offering practical, step-by-step advice to students through all stages ...
Social Research Methods
Studying a social science degree? Need to know how to develop your research methods and write up your results more effectively? In the fourth edition of this lively and engaging textbook, Alan Bryman presents students with an updated and all-encompassing guide to the principle techniques and methodology in the field of Social Research. Adopting a coherent and student-friendly format, the book ...
Bryman's Social Research Methods
Clear, comprehensive, and trusted, Bryman's Social Research Methods has guided over a quarter of a million students through their research methods course and student research project. Higher Education Skip to main ... Quantitative data analysis Part 3: Qualitative research 16:The nature of qualitative research 17:Sampling in qualitative research
The SAGE Dictionary of Social Research Methods
Written in a lively style, The SAGE Dictionary of Social Research Methods is an essential study guide for students and first-time researchers. It is a primary source of reference for advanced study, a necessary supplement to established textbooks, and a state-of-the-art reference guide to the specialized language of research across the social ...
Perspectives in Social Research Methods and Analysis
Comprising 22 reading from both academic journals and books, this text presents the best examples of research methodology in contemporary and classic sociology. The editors organized the readings according to the logic of a research project. Beginning with the research question, to design, data collection and analysis, to application in the world.
Social Science Research: Principles, Methods and Practices
This book is designed to introduce doctoral and postgraduate students to the process of conducting scientific research in the social sciences, business, education, public health, and related disciplines. It is a one-stop, comprehensive, and compact source for foundational concepts in behavioural research, and can serve as a standalone text or as a supplement to research readings in any ...
PDF The SAGE Handbook of Social Research Methods
Conducting a Meta-Analysis 536 Erika A. Patall and Harris Cooper 33. Synergy and Synthesis: Integrating Qualitative and Quantitative Data 555 Jane Fielding and Nigel Fielding ... Social Research Methods (OUP, 2001, 2004), Business Research Methods (OUP, 2003), and Disneyization of Society (SAGE, 2004). He is co-editor of The SAGE Encyclopedia ...
Social research
Social research is research conducted by social scientists following a systematic plan. Social research methodologies can be classified as quantitative and qualitative.. Quantitative designs approach social phenomena through quantifiable evidence, and often rely on statistical analyses of many cases (or across intentionally designed treatments in an experiment) to create valid and reliable ...
Research Methods for the Social Sciences: An Introduction
About This Book. Chapter 1: Introduction to Research Methods. Chapter 2: Ethics in Research. Chapter 3: Developing a Research Question. Chapter 4: Measurement and Units of Analysis. Chapter 5: The Literature Review. Chapter 6: Data Collection Strategies. Chapter 7: Sampling Techniques. Chapter 8: Data Collection Methods: Survey Research.
Sage Research Methods
Understanding Social Research brings together leading researchers in the social sciences - including sociology, health, geography, psychology and social statistics - to elaborate their approach to research design and practice, based on their own research experience, and to consider what kinds of knowledge their methods can produce. Each of the ...
Qualitative analysis
Content analysis is the systematic analysis of the content of a text—e.g., who says what, to whom, why, and to what extent and with what effect—in a quantitative or qualitative manner. Content analysis is typically conducted as follows. First, when there are many texts to analyse—e.g., newspaper stories, financial reports, blog postings ...
Learning to Do Qualitative Data Analysis: A Starting Point
For many researchers unfamiliar with qualitative research, determining how to conduct qualitative analyses is often quite challenging. Part of this challenge is due to the seemingly limitless approaches that a qualitative researcher might leverage, as well as simply learning to think like a qualitative researcher when analyzing data. From framework analysis (Ritchie & Spencer, 1994) to content ...
Deductive Qualitative Analysis: Evaluating, Expanding, and Refining
Deductive qualitative analysis (DQA; Gilgun, 2005) is a specific approach to deductive qualitative research intended to systematically test, refine, or refute theory by integrating deductive and inductive strands of inquiry.The purpose of the present paper is to provide a primer on the basic principles and practices of DQA and to exemplify the methodology using two studies that were conducted ...
Research Methods
Research methods are specific procedures for collecting and analyzing data. Developing your research methods is an integral part of your research design. When planning your methods, there are two key decisions you will make. First, decide how you will collect data. Your methods depend on what type of data you need to answer your research question:
SAGE Research Methods: Find resources to answer your research methods
Click to continue
Social Research
Social Research is a method used by social scientists and researchers to learn about people and societies so that they can design products/services that cater to various needs of the people. Learn about the definition of social research along with its types and methods. ... Quantitative methods use statistical analysis techniques to evaluate ...
Sociological Methods & Research: Sage Journals
When your research depends on the very latest information on the collection, measurement and analysis of data, turn to Sociological Methods & Research (SMR).Each issue of SMR presents new techniques and innovative approaches to recurring research challenges and clarifies existing methods. The journal also provides state-of-the-art tools that researchers and academics need to increase the ...
Social Research Methods: Explanation and Examples
Social research methods provide us with a framework to investigate the complexities of human societies. By utilizing various techniques such as surveys, interviews, observations, experiments, and content analysis, we gain insights into human behavior and social interactions that guide improvements in our communities and policies.
Social Data Analysis
Social data analysis enables you, as a researcher, to organize the facts you collect during your research. Your data may have come from a questionnaire survey, a set of interviews, or observations. They may be data that have been made available to you from some organization, national or international agency or other researchers. Whatever their source, social data can be daunting to put ...
A systematic review of Stimulated Recall (SR) in educational research
For example, some articles merely reviewed others' research on the SR method employed in teaching settings or using painting-based stimuli to spark students' prior knowledge did not meet the ...
A qualitative study of rural healthcare providers' views of social
Qualitative content analysis was used, a method particularly beneficial for analyzing large amounts of qualitative ... Waters MC. Flexible Coding of In-depth Interviews: A Twenty-first-century Approach. Sociological Methods & Research. 2021;50(2):708-39. ... InForum Qualitative Sozialforschung/Forum: Qualitative Social Research. 2019;20:26 ...
Kristen Kennedy Terry and Bob Bayley Publish New Book Titled: "Social
Congratulations to Dr. Kristen Kennedy Terry (UC Santa Cruz) and Dr. Bob Bayley (UC Davis) for publishing "Social Network Analysis in Second Language Research: Theory and Methods"! Click the link below for more information.
Replicating the Job Importance and Job Satisfaction Latent Class
Replication or reproducibility of results is a cornerstone of scientific research, as replication studies can identify artifacts that affect internal validity, investigate sampling error, increase generalizability, provide further testing of the original hypothesis, and evaluate claims of fraud. The purpose of the current working paper is to determine whether the five-class, three-response ...
Modelling
Construction Engineering and Management (CEM) is a broad domain with publications covering interrelated subdisciplines and considered a key source of knowledge sharing. Previous studies used scientometric methods to assess the current impact of CEM publications; however, there is a need to predict future citations of CEM publications to identify the expected high-impact trends in the future ...

2.2 Research Methods

Field Research

Sociology in the Real World

Participant Observation

Ethnography

Institutional Ethnography

Sociological Research

Experiments

You are here

Perspectives in Social Research Methods and Analysis A Reader for Sociology

Sample Materials & Chapters

For instructors

13 Qualitative analysis

Grounded theory

Content analysis

Hermeneutic analysis

Share This Book

Have a language expert improve your writing

Research Methods | Definitions, Types, Examples

Table of contents

Qualitative vs. quantitative data

Primary vs. secondary research

Descriptive vs. experimental data

Prevent plagiarism. Run a free check.

Qualitative analysis methods

Quantitative analysis methods

Here's why students love Scribbr's proofreading services

Is this article helpful?

More interesting articles

Unlimited Academic AI-Proofreading

Social Research – Definition, Types and Methods

Social Research: Definition

Types of Social Research

Social Research Methods

MORE LIKE THIS

Top 17 UX Research Software for UX Design in 2024

Healthcare Staff Burnout: What it Is + How To Manage It

Top 15 Employee Retention Software in 2024

Top 10 Employee Development Software for Talent Growth

Other categories

Social Research Methods

Types of Social Research Methods

Examples of Social Research Methods

Why is it Important?

Origin of Social Research Methods

Controversies in Social Research

Related Topics

Social Data Analysis

Formats Available

Table of Contents

Ancillary Material

About the Contributors

Contribute to this Page

A systematic review of Stimulated Recall (SR) in educational research from 2012 to 2022

Similar content being viewed by others

The science of effective learning with spacing and retrieval practice

A meta-analysis to gauge the impact of pedagogies employed in mixed-ability high school biology classrooms

Preservice teachers’ experiences of observing their teaching competencies via self-recorded videos in a personalized learning environment

Introduction

Literature review

The educational application using SR

S timuli and interview strategies in SR

Paper selection

Coding procedure

Results and discussion

RQ1: Research aims

Exploring thought patterns

Investigating the effect of educational strategies

Improving metacognition

RQ2: Stimuli

Optimizing video stimuli

Utilizing biofeedback data

RQ3: Time factors

Enhancing the temporal properties of stimuli

Setting up flexible intervals

RQ4: Interview strategies

Posing appropriate questions

Staying value-neutral in guiding

RQ5: The relationship between different coding items

The relationship between research aims and learning subject