problem solving learning system

Center for Teaching Innovation

Resource library.

• Establishing Community Agreements and Classroom Norms
• Sample group work rubric
• Problem-Based Learning Clearinghouse of Activities, University of Delaware

Problem-Based Learning

Problem-based learning  (PBL) is a student-centered approach in which students learn about a subject by working in groups to solve an open-ended problem. This problem is what drives the motivation and the learning. 

Why Use Problem-Based Learning?

Nilson (2010) lists the following learning outcomes that are associated with PBL. A well-designed PBL project provides students with the opportunity to develop skills related to:

• Working in teams.
• Managing projects and holding leadership roles.
• Oral and written communication.
• Self-awareness and evaluation of group processes.
• Working independently.
• Critical thinking and analysis.
• Explaining concepts.
• Self-directed learning.
• Applying course content to real-world examples.
• Researching and information literacy.
• Problem solving across disciplines.

Considerations for Using Problem-Based Learning

Rather than teaching relevant material and subsequently having students apply the knowledge to solve problems, the problem is presented first. PBL assignments can be short, or they can be more involved and take a whole semester. PBL is often group-oriented, so it is beneficial to set aside classroom time to prepare students to   work in groups  and to allow them to engage in their PBL project.

Students generally must:

• Examine and define the problem.
• Explore what they already know about underlying issues related to it.
• Determine what they need to learn and where they can acquire the information and tools necessary to solve the problem.
• Evaluate possible ways to solve the problem.
• Solve the problem.
• Report on their findings.

Getting Started with Problem-Based Learning

• Articulate the learning outcomes of the project. What do you want students to know or be able to do as a result of participating in the assignment?
• Create the problem. Ideally, this will be a real-world situation that resembles something students may encounter in their future careers or lives. Cases are often the basis of PBL activities. Previously developed PBL activities can be found online through the University of Delaware’s PBL Clearinghouse of Activities .
• Establish ground rules at the beginning to prepare students to work effectively in groups.
• Introduce students to group processes and do some warm up exercises to allow them to practice assessing both their own work and that of their peers.
• Consider having students take on different roles or divide up the work up amongst themselves. Alternatively, the project might require students to assume various perspectives, such as those of government officials, local business owners, etc.
• Establish how you will evaluate and assess the assignment. Consider making the self and peer assessments a part of the assignment grade.

Nilson, L. B. (2010).  Teaching at its best: A research-based resource for college instructors  (2nd ed.).  San Francisco, CA: Jossey-Bass. 

Learn by doing

Guided interactive problem solving that’s effective and fun. master concepts in 15 minutes a day., data analysis, computer science, programming & ai, science & engineering, join over 10 million people learning on brilliant, over 50,000 5-star reviews on ios app store and google play.

Master concepts in 15 minutes a day

Whether you’re a complete beginner or ready to dive into machine learning and beyond, Brilliant makes it easy to level up fast with fun, bite-sized lessons.

Effective, hands-on learning

Visual, interactive lessons make concepts feel intuitive — so even complex ideas just click. Our real-time feedback and simple explanations make learning efficient.

Learn at your level

Students and professionals alike can hone dormant skills or learn new ones. Progress through lessons and challenges tailored to your level. Designed for ages 13 to 113.

Guided bite-sized lessons

We make it easy to stay on track, see your progress, and build your problem solving skills one concept at a time.

Stay motivated

Form a real learning habit with fun content that’s always well-paced, game-like progress tracking, and friendly reminders.

Guided courses for every journey

All of our courses are crafted by award-winning teachers, researchers, and professionals from MIT, Caltech, Duke, Microsoft, Google, and more.

• Foundational Math
• Software Development
• Foundational Logic
• Data Science
• High School Math
• Engineering
• Statistics and Finance

Courses in Foundational Math

• Solving Equations
• Measuring with Geometry
• Mathematical Fundamentals
• Reasoning with Algebra
• Functions and Quadratics

10k+ Ratings

60k+ Ratings

We use cookies to improve your experience on Brilliant. Learn more about our cookie policy and settings .

Problem-Based Learning (PBL)

What is Problem-Based Learning (PBL)? PBL is a student-centered approach to learning that involves groups of students working to solve a real-world problem, quite different from the direct teaching method of a teacher presenting facts and concepts about a specific subject to a classroom of students. Through PBL, students not only strengthen their teamwork, communication, and research skills, but they also sharpen their critical thinking and problem-solving abilities essential for life-long learning.

See also: Just-in-Time Teaching

In implementing PBL, the teaching role shifts from that of the more traditional model that follows a linear, sequential pattern where the teacher presents relevant material, informs the class what needs to be done, and provides details and information for students to apply their knowledge to a given problem. With PBL, the teacher acts as a facilitator; the learning is student-driven with the aim of solving the given problem (note: the problem is established at the onset of learning opposed to being presented last in the traditional model). Also, the assignments vary in length from relatively short to an entire semester with daily instructional time structured for group work.

By working with PBL, students will:

• Become engaged with open-ended situations that assimilate the world of work
• Participate in groups to pinpoint what is known/ not known and the methods of finding information to help solve the given problem.
• Investigate a problem; through critical thinking and problem solving, brainstorm a list of unique solutions.
• Analyze the situation to see if the real problem is framed or if there are other problems that need to be solved.

How to Begin PBL

• Establish the learning outcomes (i.e., what is it that you want your students to really learn and to be able to do after completing the learning project).
• Find a real-world problem that is relevant to the students; often the problems are ones that students may encounter in their own life or future career.
• Discuss pertinent rules for working in groups to maximize learning success.
• Practice group processes: listening, involving others, assessing their work/peers.
• Explore different roles for students to accomplish the work that needs to be done and/or to see the problem from various perspectives depending on the problem (e.g., for a problem about pollution, different roles may be a mayor, business owner, parent, child, neighboring city government officials, etc.).
• Determine how the project will be evaluated and assessed. Most likely, both self-assessment and peer-assessment will factor into the assignment grade.

Designing Classroom Instruction

See also: Inclusive Teaching Strategies

• Take the curriculum and divide it into various units. Decide on the types of problems that your students will solve. These will be your objectives.
• Determine the specific problems that most likely have several answers; consider student interest.
• Arrange appropriate resources available to students; utilize other teaching personnel to support students where needed (e.g., media specialists to orientate students to electronic references).
• Decide on presentation formats to communicate learning (e.g., individual paper, group PowerPoint, an online blog, etc.) and appropriate grading mechanisms (e.g., rubric).
• Decide how to incorporate group participation (e.g., what percent, possible peer evaluation, etc.).

How to Orchestrate a PBL Activity

• Explain Problem-Based Learning to students: its rationale, daily instruction, class expectations, grading.
• Serve as a model and resource to the PBL process; work in-tandem through the first problem
• Help students secure various resources when needed.
• Supply ample class time for collaborative group work.
• Give feedback to each group after they share via the established format; critique the solution in quality and thoroughness. Reinforce to the students that the prior thinking and reasoning process in addition to the solution are important as well.

Teacher’s Role in PBL

See also: Flipped teaching

As previously mentioned, the teacher determines a problem that is interesting, relevant, and novel for the students. It also must be multi-faceted enough to engage students in doing research and finding several solutions. The problems stem from the unit curriculum and reflect possible use in future work situations.

• Determine a problem aligned with the course and your students. The problem needs to be demanding enough that the students most likely cannot solve it on their own. It also needs to teach them new skills. When sharing the problem with students, state it in a narrative complete with pertinent background information without excessive information. Allow the students to find out more details as they work on the problem.
• Place students in groups, well-mixed in diversity and skill levels, to strengthen the groups. Help students work successfully. One way is to have the students take on various roles in the group process after they self-assess their strengths and weaknesses.
• Support the students with understanding the content on a deeper level and in ways to best orchestrate the various stages of the problem-solving process.

The Role of the Students

See also: ADDIE model

The students work collaboratively on all facets of the problem to determine the best possible solution.

• Analyze the problem and the issues it presents. Break the problem down into various parts. Continue to read, discuss, and think about the problem.
• Construct a list of what is known about the problem. What do your fellow students know about the problem? Do they have any experiences related to the problem? Discuss the contributions expected from the team members. What are their strengths and weaknesses? Follow the rules of brainstorming (i.e., accept all answers without passing judgment) to generate possible solutions for the problem.
• Get agreement from the team members regarding the problem statement.
• Put the problem statement in written form.
• Solicit feedback from the teacher.
• Be open to changing the written statement based on any new learning that is found or feedback provided.
• Generate a list of possible solutions. Include relevant thoughts, ideas, and educated guesses as well as causes and possible ways to solve it. Then rank the solutions and select the solution that your group is most likely to perceive as the best in terms of meeting success.
• Include what needs to be known and done to solve the identified problems.
• Prioritize the various action steps.
• Consider how the steps impact the possible solutions.
• See if the group is in agreement with the timeline; if not, decide how to reach agreement.
• What resources are available to help (e.g., textbooks, primary/secondary sources, Internet).
• Determine research assignments per team members.
• Establish due dates.
• Determine how your group will present the problem solution and also identify the audience. Usually, in PBL, each group presents their solutions via a team presentation either to the class of other students or to those who are related to the problem.
• Both the process and the results of the learning activity need to be covered. Include the following: problem statement, questions, data gathered, data analysis, reasons for the solution(s) and/or any recommendations reflective of the data analysis.
• A well-stated problem and conclusion.
• The process undertaken by the group in solving the problem, the various options discussed, and the resources used.
• Your solution’s supporting documents, guests, interviews and their purpose to be convincing to your audience.
• In addition, be prepared for any audience comments and questions. Determine who will respond and if your team doesn’t know the answer, admit this and be open to looking into the question at a later date.
• Reflective thinking and transfer of knowledge are important components of PBL. This helps the students be more cognizant of their own learning and teaches them how to ask appropriate questions to address problems that need to be solved. It is important to look at both the individual student and the group effort/delivery throughout the entire process. From here, you can better determine what was learned and how to improve. The students should be asked how they can apply what was learned to a different situation, to their own lives, and to other course projects.

See also: Kirkpatrick Model: Four Levels of Learning Evaluation

I am a professor of Educational Technology. I have worked at several elite universities. I hold a PhD degree from the University of Illinois and a master's degree from Purdue University.

Similar Posts

Definitions of the addie model.

What is the ADDIE Model? This article attempts to explain the ADDIE model by providing different definitions. Basically, ADDIE is a conceptual framework. ADDIE is the most commonly used instructional design framework and…

Robert Gagné’s Taxonomy of Learning

Gagne classified learning outcomes into five major categories: verbal information, intellectual skills, cognitive strategies, motor skills and attitudes. What is learning to Gagné? As outlined in his 9-events of instruction, Gagne believed that learning was…

Open Source Learning Management Systems (LMS)

Learning Management Systems (LMSs) are becoming a vital part of classrooms in the 21th Century. This is a list of open source learning management systems. By open source we mean that source code of…

Adaptive Learning: What is It, What are its Benefits and How Does it Work?

People learn in many different ways. Adaptive learning has sought to address differences in ability by targeting teaching practices. The use of adaptive models, ranging from technological programs to intelligent systems, can be…

Instructional Design

What is Instructional Design The Association for Educational Communications and Technology (AECT) defines instructional design as “the theory and practice of design, development, utilization, management, and evaluation of processes and resources for learning”…

Concept Maps and How To Use Them

Concept maps help our brains take in information, mostly when there is visual information. The maps help us to see the big picture along with the connected and related data. They also help…

• Illinois Online
• Illinois Remote

• TA Resources
• Teaching Consultation
• Teaching Portfolio Program
• Grad Academy for College Teaching
• Faculty Events
• The Art of Teaching
• 2022 Illinois Summer Teaching Institute
• Large Classes
• Leading Discussions
• Laboratory Classes
• Lecture-Based Classes
• Planning a Class Session
• Questioning Strategies
• Classroom Assessment Techniques (CATs)
• Problem-Based Learning (PBL)
• The Case Method
• Community-Based Learning: Service Learning
• Group Learning
• Just-in-Time Teaching
• Creating a Syllabus
• Motivating Students
• Dealing With Cheating
• Discouraging & Detecting Plagiarism
• Diversity & Creating an Inclusive Classroom
• Harassment & Discrimination
• Professional Conduct
• Foundations of Good Teaching
• Student Engagement
• Assessment Strategies
• Course Design
• Student Resources
• Teaching Tips
• Graduate Teacher Certificate
• Certificate in Foundations of Teaching
• Teacher Scholar Certificate
• Certificate in Technology-Enhanced Teaching
• Master Course in Online Teaching (MCOT)
• 2022 Celebration of College Teaching
• 2023 Celebration of College Teaching
• Hybrid Teaching and Learning Certificate
• 2024 Celebration of College Teaching
• Classroom Observation Etiquette
• Teaching Philosophy Statement
• Pedagogical Literature Review
• Scholarship of Teaching and Learning
• Instructor Stories
• Podcast: Teach Talk Listen Learn
• Universal Design for Learning

Sign-Up to receive Teaching and Learning news and events

Problem-Based Learning (PBL) is a teaching method in which complex real-world problems are used as the vehicle to promote student learning of concepts and principles as opposed to direct presentation of facts and concepts. In addition to course content, PBL can promote the development of critical thinking skills, problem-solving abilities, and communication skills. It can also provide opportunities for working in groups, finding and evaluating research materials, and life-long learning (Duch et al, 2001).

PBL can be incorporated into any learning situation. In the strictest definition of PBL, the approach is used over the entire semester as the primary method of teaching. However, broader definitions and uses range from including PBL in lab and design classes, to using it simply to start a single discussion. PBL can also be used to create assessment items. The main thread connecting these various uses is the real-world problem.

Any subject area can be adapted to PBL with a little creativity. While the core problems will vary among disciplines, there are some characteristics of good PBL problems that transcend fields (Duch, Groh, and Allen, 2001):

• The problem must motivate students to seek out a deeper understanding of concepts.
• The problem should require students to make reasoned decisions and to defend them.
• The problem should incorporate the content objectives in such a way as to connect it to previous courses/knowledge.
• If used for a group project, the problem needs a level of complexity to ensure that the students must work together to solve it.
• If used for a multistage project, the initial steps of the problem should be open-ended and engaging to draw students into the problem.

The problems can come from a variety of sources: newspapers, magazines, journals, books, textbooks, and television/ movies. Some are in such form that they can be used with little editing; however, others need to be rewritten to be of use. The following guidelines from The Power of Problem-Based Learning (Duch et al, 2001) are written for creating PBL problems for a class centered around the method; however, the general ideas can be applied in simpler uses of PBL:

• Choose a central idea, concept, or principle that is always taught in a given course, and then think of a typical end-of-chapter problem, assignment, or homework that is usually assigned to students to help them learn that concept. List the learning objectives that students should meet when they work through the problem.
• Think of a real-world context for the concept under consideration. Develop a storytelling aspect to an end-of-chapter problem, or research an actual case that can be adapted, adding some motivation for students to solve the problem. More complex problems will challenge students to go beyond simple plug-and-chug to solve it. Look at magazines, newspapers, and articles for ideas on the story line. Some PBL practitioners talk to professionals in the field, searching for ideas of realistic applications of the concept being taught.
• What will the first page (or stage) look like? What open-ended questions can be asked? What learning issues will be identified?
• How will the problem be structured?
• How long will the problem be? How many class periods will it take to complete?
• Will students be given information in subsequent pages (or stages) as they work through the problem?
• What resources will the students need?
• What end product will the students produce at the completion of the problem?
• Write a teacher's guide detailing the instructional plans on using the problem in the course. If the course is a medium- to large-size class, a combination of mini-lectures, whole-class discussions, and small group work with regular reporting may be necessary. The teacher's guide can indicate plans or options for cycling through the pages of the problem interspersing the various modes of learning.
• The final step is to identify key resources for students. Students need to learn to identify and utilize learning resources on their own, but it can be helpful if the instructor indicates a few good sources to get them started. Many students will want to limit their research to the Internet, so it will be important to guide them toward the library as well.

The method for distributing a PBL problem falls under three closely related teaching techniques: case studies, role-plays, and simulations. Case studies are presented to students in written form. Role-plays have students improvise scenes based on character descriptions given. Today, simulations often involve computer-based programs. Regardless of which technique is used, the heart of the method remains the same: the real-world problem.

Where can I learn more?

• PBL through the Institute for Transforming Undergraduate Education at the University of Delaware
• Duch, B. J., Groh, S. E, & Allen, D. E. (Eds.). (2001). The power of problem-based learning . Sterling, VA: Stylus.
• Grasha, A. F. (1996). Teaching with style: A practical guide to enhancing learning by understanding teaching and learning styles. Pittsburgh: Alliance Publishers.

Center for Innovation in Teaching & Learning

249 Armory Building 505 East Armory Avenue Champaign, IL 61820

217 333-1462

Email: [email protected]

Office of the Provost

Center for Teaching

Teaching problem solving.

Print Version

Tips and Techniques

Expert vs. novice problem solvers, communicate.

• Have students  identify specific problems, difficulties, or confusions . Don’t waste time working through problems that students already understand.
• If students are unable to articulate their concerns, determine where they are having trouble by  asking them to identify the specific concepts or principles associated with the problem.
• In a one-on-one tutoring session, ask the student to  work his/her problem out loud . This slows down the thinking process, making it more accurate and allowing you to access understanding.
• When working with larger groups you can ask students to provide a written “two-column solution.” Have students write up their solution to a problem by putting all their calculations in one column and all of their reasoning (in complete sentences) in the other column. This helps them to think critically about their own problem solving and helps you to more easily identify where they may be having problems. Two-Column Solution (Math) Two-Column Solution (Physics)

Encourage Independence

• Model the problem solving process rather than just giving students the answer. As you work through the problem, consider how a novice might struggle with the concepts and make your thinking clear
• Have students work through problems on their own. Ask directing questions or give helpful suggestions, but  provide only minimal assistance and only when needed to overcome obstacles.
• Don’t fear  group work ! Students can frequently help each other, and talking about a problem helps them think more critically about the steps needed to solve the problem. Additionally, group work helps students realize that problems often have multiple solution strategies, some that might be more effective than others

Be sensitive

• Frequently, when working problems, students are unsure of themselves. This lack of confidence may hamper their learning. It is important to recognize this when students come to us for help, and to give each student some feeling of mastery. Do this by providing  positive reinforcement to let students know when they have mastered a new concept or skill.

Encourage Thoroughness and Patience

• Try to communicate that  the process is more important than the answer so that the student learns that it is OK to not have an instant solution. This is learned through your acceptance of his/her pace of doing things, through your refusal to let anxiety pressure you into giving the right answer, and through your example of problem solving through a step-by step process.

Experts (teachers) in a particular field are often so fluent in solving problems from that field that they can find it difficult to articulate the problem solving principles and strategies they use to novices (students) in their field because these principles and strategies are second nature to the expert. To teach students problem solving skills,  a teacher should be aware of principles and strategies of good problem solving in his or her discipline .

The mathematician George Polya captured the problem solving principles and strategies he used in his discipline in the book  How to Solve It: A New Aspect of Mathematical Method (Princeton University Press, 1957). The book includes  a summary of Polya’s problem solving heuristic as well as advice on the teaching of problem solving.

Teaching Guides

• Online Course Development Resources
• Principles & Frameworks
• Pedagogies & Strategies
• Reflecting & Assessing
• Challenges & Opportunities
• Populations & Contexts

Quick Links

• Services for Departments and Schools
• Examples of Online Instructional Modules

How to master the seven-step problem-solving process

In this episode of the McKinsey Podcast , Simon London speaks with Charles Conn, CEO of venture-capital firm Oxford Sciences Innovation, and McKinsey senior partner Hugo Sarrazin about the complexities of different problem-solving strategies.

Podcast transcript

Simon London: Hello, and welcome to this episode of the McKinsey Podcast , with me, Simon London. What’s the number-one skill you need to succeed professionally? Salesmanship, perhaps? Or a facility with statistics? Or maybe the ability to communicate crisply and clearly? Many would argue that at the very top of the list comes problem solving: that is, the ability to think through and come up with an optimal course of action to address any complex challenge—in business, in public policy, or indeed in life.

Looked at this way, it’s no surprise that McKinsey takes problem solving very seriously, testing for it during the recruiting process and then honing it, in McKinsey consultants, through immersion in a structured seven-step method. To discuss the art of problem solving, I sat down in California with McKinsey senior partner Hugo Sarrazin and also with Charles Conn. Charles is a former McKinsey partner, entrepreneur, executive, and coauthor of the book Bulletproof Problem Solving: The One Skill That Changes Everything [John Wiley & Sons, 2018].

Charles and Hugo, welcome to the podcast. Thank you for being here.

Hugo Sarrazin: Our pleasure.

Charles Conn: It’s terrific to be here.

Simon London: Problem solving is a really interesting piece of terminology. It could mean so many different things. I have a son who’s a teenage climber. They talk about solving problems. Climbing is problem solving. Charles, when you talk about problem solving, what are you talking about?

Charles Conn: For me, problem solving is the answer to the question “What should I do?” It’s interesting when there’s uncertainty and complexity, and when it’s meaningful because there are consequences. Your son’s climbing is a perfect example. There are consequences, and it’s complicated, and there’s uncertainty—can he make that grab? I think we can apply that same frame almost at any level. You can think about questions like “What town would I like to live in?” or “Should I put solar panels on my roof?”

You might think that’s a funny thing to apply problem solving to, but in my mind it’s not fundamentally different from business problem solving, which answers the question “What should my strategy be?” Or problem solving at the policy level: “How do we combat climate change?” “Should I support the local school bond?” I think these are all part and parcel of the same type of question, “What should I do?”

I’m a big fan of structured problem solving. By following steps, we can more clearly understand what problem it is we’re solving, what are the components of the problem that we’re solving, which components are the most important ones for us to pay attention to, which analytic techniques we should apply to those, and how we can synthesize what we’ve learned back into a compelling story. That’s all it is, at its heart.

I think sometimes when people think about seven steps, they assume that there’s a rigidity to this. That’s not it at all. It’s actually to give you the scope for creativity, which often doesn’t exist when your problem solving is muddled.

Simon London: You were just talking about the seven-step process. That’s what’s written down in the book, but it’s a very McKinsey process as well. Without getting too deep into the weeds, let’s go through the steps, one by one. You were just talking about problem definition as being a particularly important thing to get right first. That’s the first step. Hugo, tell us about that.

Hugo Sarrazin: It is surprising how often people jump past this step and make a bunch of assumptions. The most powerful thing is to step back and ask the basic questions—“What are we trying to solve? What are the constraints that exist? What are the dependencies?” Let’s make those explicit and really push the thinking and defining. At McKinsey, we spend an enormous amount of time in writing that little statement, and the statement, if you’re a logic purist, is great. You debate. “Is it an ‘or’? Is it an ‘and’? What’s the action verb?” Because all these specific words help you get to the heart of what matters.

Want to subscribe to The McKinsey Podcast ?

Simon London: So this is a concise problem statement.

Hugo Sarrazin: Yeah. It’s not like “Can we grow in Japan?” That’s interesting, but it is “What, specifically, are we trying to uncover in the growth of a product in Japan? Or a segment in Japan? Or a channel in Japan?” When you spend an enormous amount of time, in the first meeting of the different stakeholders, debating this and having different people put forward what they think the problem definition is, you realize that people have completely different views of why they’re here. That, to me, is the most important step.

Charles Conn: I would agree with that. For me, the problem context is critical. When we understand “What are the forces acting upon your decision maker? How quickly is the answer needed? With what precision is the answer needed? Are there areas that are off limits or areas where we would particularly like to find our solution? Is the decision maker open to exploring other areas?” then you not only become more efficient, and move toward what we call the critical path in problem solving, but you also make it so much more likely that you’re not going to waste your time or your decision maker’s time.

How often do especially bright young people run off with half of the idea about what the problem is and start collecting data and start building models—only to discover that they’ve really gone off half-cocked.

Hugo Sarrazin: Yeah.

Charles Conn: And in the wrong direction.

Simon London: OK. So step one—and there is a real art and a structure to it—is define the problem. Step two, Charles?

Charles Conn: My favorite step is step two, which is to use logic trees to disaggregate the problem. Every problem we’re solving has some complexity and some uncertainty in it. The only way that we can really get our team working on the problem is to take the problem apart into logical pieces.

What we find, of course, is that the way to disaggregate the problem often gives you an insight into the answer to the problem quite quickly. I love to do two or three different cuts at it, each one giving a bit of a different insight into what might be going wrong. By doing sensible disaggregations, using logic trees, we can figure out which parts of the problem we should be looking at, and we can assign those different parts to team members.

Simon London: What’s a good example of a logic tree on a sort of ratable problem?

Charles Conn: Maybe the easiest one is the classic profit tree. Almost in every business that I would take a look at, I would start with a profit or return-on-assets tree. In its simplest form, you have the components of revenue, which are price and quantity, and the components of cost, which are cost and quantity. Each of those can be broken out. Cost can be broken into variable cost and fixed cost. The components of price can be broken into what your pricing scheme is. That simple tree often provides insight into what’s going on in a business or what the difference is between that business and the competitors.

If we add the leg, which is “What’s the asset base or investment element?”—so profit divided by assets—then we can ask the question “Is the business using its investments sensibly?” whether that’s in stores or in manufacturing or in transportation assets. I hope we can see just how simple this is, even though we’re describing it in words.

When I went to work with Gordon Moore at the Moore Foundation, the problem that he asked us to look at was “How can we save Pacific salmon?” Now, that sounds like an impossible question, but it was amenable to precisely the same type of disaggregation and allowed us to organize what became a 15-year effort to improve the likelihood of good outcomes for Pacific salmon.

Simon London: Now, is there a danger that your logic tree can be impossibly large? This, I think, brings us onto the third step in the process, which is that you have to prioritize.

Charles Conn: Absolutely. The third step, which we also emphasize, along with good problem definition, is rigorous prioritization—we ask the questions “How important is this lever or this branch of the tree in the overall outcome that we seek to achieve? How much can I move that lever?” Obviously, we try and focus our efforts on ones that have a big impact on the problem and the ones that we have the ability to change. With salmon, ocean conditions turned out to be a big lever, but not one that we could adjust. We focused our attention on fish habitats and fish-harvesting practices, which were big levers that we could affect.

People spend a lot of time arguing about branches that are either not important or that none of us can change. We see it in the public square. When we deal with questions at the policy level—“Should you support the death penalty?” “How do we affect climate change?” “How can we uncover the causes and address homelessness?”—it’s even more important that we’re focusing on levers that are big and movable.

Would you like to learn more about our Strategy & Corporate Finance Practice ?

Simon London: Let’s move swiftly on to step four. You’ve defined your problem, you disaggregate it, you prioritize where you want to analyze—what you want to really look at hard. Then you got to the work plan. Now, what does that mean in practice?

Hugo Sarrazin: Depending on what you’ve prioritized, there are many things you could do. It could be breaking the work among the team members so that people have a clear piece of the work to do. It could be defining the specific analyses that need to get done and executed, and being clear on time lines. There’s always a level-one answer, there’s a level-two answer, there’s a level-three answer. Without being too flippant, I can solve any problem during a good dinner with wine. It won’t have a whole lot of backing.

Simon London: Not going to have a lot of depth to it.

Hugo Sarrazin: No, but it may be useful as a starting point. If the stakes are not that high, that could be OK. If it’s really high stakes, you may need level three and have the whole model validated in three different ways. You need to find a work plan that reflects the level of precision, the time frame you have, and the stakeholders you need to bring along in the exercise.

Charles Conn: I love the way you’ve described that, because, again, some people think of problem solving as a linear thing, but of course what’s critical is that it’s iterative. As you say, you can solve the problem in one day or even one hour.

Charles Conn: We encourage our teams everywhere to do that. We call it the one-day answer or the one-hour answer. In work planning, we’re always iterating. Every time you see a 50-page work plan that stretches out to three months, you know it’s wrong. It will be outmoded very quickly by that learning process that you described. Iterative problem solving is a critical part of this. Sometimes, people think work planning sounds dull, but it isn’t. It’s how we know what’s expected of us and when we need to deliver it and how we’re progressing toward the answer. It’s also the place where we can deal with biases. Bias is a feature of every human decision-making process. If we design our team interactions intelligently, we can avoid the worst sort of biases.

Simon London: Here we’re talking about cognitive biases primarily, right? It’s not that I’m biased against you because of your accent or something. These are the cognitive biases that behavioral sciences have shown we all carry around, things like anchoring, overoptimism—these kinds of things.

Both: Yeah.

Charles Conn: Availability bias is the one that I’m always alert to. You think you’ve seen the problem before, and therefore what’s available is your previous conception of it—and we have to be most careful about that. In any human setting, we also have to be careful about biases that are based on hierarchies, sometimes called sunflower bias. I’m sure, Hugo, with your teams, you make sure that the youngest team members speak first. Not the oldest team members, because it’s easy for people to look at who’s senior and alter their own creative approaches.

Hugo Sarrazin: It’s helpful, at that moment—if someone is asserting a point of view—to ask the question “This was true in what context?” You’re trying to apply something that worked in one context to a different one. That can be deadly if the context has changed, and that’s why organizations struggle to change. You promote all these people because they did something that worked well in the past, and then there’s a disruption in the industry, and they keep doing what got them promoted even though the context has changed.

Simon London: Right. Right.

Hugo Sarrazin: So it’s the same thing in problem solving.

Charles Conn: And it’s why diversity in our teams is so important. It’s one of the best things about the world that we’re in now. We’re likely to have people from different socioeconomic, ethnic, and national backgrounds, each of whom sees problems from a slightly different perspective. It is therefore much more likely that the team will uncover a truly creative and clever approach to problem solving.

Simon London: Let’s move on to step five. You’ve done your work plan. Now you’ve actually got to do the analysis. The thing that strikes me here is that the range of tools that we have at our disposal now, of course, is just huge, particularly with advances in computation, advanced analytics. There’s so many things that you can apply here. Just talk about the analysis stage. How do you pick the right tools?

Charles Conn: For me, the most important thing is that we start with simple heuristics and explanatory statistics before we go off and use the big-gun tools. We need to understand the shape and scope of our problem before we start applying these massive and complex analytical approaches.

Simon London: Would you agree with that?

Hugo Sarrazin: I agree. I think there are so many wonderful heuristics. You need to start there before you go deep into the modeling exercise. There’s an interesting dynamic that’s happening, though. In some cases, for some types of problems, it is even better to set yourself up to maximize your learning. Your problem-solving methodology is test and learn, test and learn, test and learn, and iterate. That is a heuristic in itself, the A/B testing that is used in many parts of the world. So that’s a problem-solving methodology. It’s nothing different. It just uses technology and feedback loops in a fast way. The other one is exploratory data analysis. When you’re dealing with a large-scale problem, and there’s so much data, I can get to the heuristics that Charles was talking about through very clever visualization of data.

You test with your data. You need to set up an environment to do so, but don’t get caught up in neural-network modeling immediately. You’re testing, you’re checking—“Is the data right? Is it sound? Does it make sense?”—before you launch too far.

Simon London: You do hear these ideas—that if you have a big enough data set and enough algorithms, they’re going to find things that you just wouldn’t have spotted, find solutions that maybe you wouldn’t have thought of. Does machine learning sort of revolutionize the problem-solving process? Or are these actually just other tools in the toolbox for structured problem solving?

Charles Conn: It can be revolutionary. There are some areas in which the pattern recognition of large data sets and good algorithms can help us see things that we otherwise couldn’t see. But I do think it’s terribly important we don’t think that this particular technique is a substitute for superb problem solving, starting with good problem definition. Many people use machine learning without understanding algorithms that themselves can have biases built into them. Just as 20 years ago, when we were doing statistical analysis, we knew that we needed good model definition, we still need a good understanding of our algorithms and really good problem definition before we launch off into big data sets and unknown algorithms.

Simon London: Step six. You’ve done your analysis.

Charles Conn: I take six and seven together, and this is the place where young problem solvers often make a mistake. They’ve got their analysis, and they assume that’s the answer, and of course it isn’t the answer. The ability to synthesize the pieces that came out of the analysis and begin to weave those into a story that helps people answer the question “What should I do?” This is back to where we started. If we can’t synthesize, and we can’t tell a story, then our decision maker can’t find the answer to “What should I do?”

Simon London: But, again, these final steps are about motivating people to action, right?

Charles Conn: Yeah.

Simon London: I am slightly torn about the nomenclature of problem solving because it’s on paper, right? Until you motivate people to action, you actually haven’t solved anything.

Charles Conn: I love this question because I think decision-making theory, without a bias to action, is a waste of time. Everything in how I approach this is to help people take action that makes the world better.

Simon London: Hence, these are absolutely critical steps. If you don’t do this well, you’ve just got a bunch of analysis.

Charles Conn: We end up in exactly the same place where we started, which is people speaking across each other, past each other in the public square, rather than actually working together, shoulder to shoulder, to crack these important problems.

Simon London: In the real world, we have a lot of uncertainty—arguably, increasing uncertainty. How do good problem solvers deal with that?

Hugo Sarrazin: At every step of the process. In the problem definition, when you’re defining the context, you need to understand those sources of uncertainty and whether they’re important or not important. It becomes important in the definition of the tree.

You need to think carefully about the branches of the tree that are more certain and less certain as you define them. They don’t have equal weight just because they’ve got equal space on the page. Then, when you’re prioritizing, your prioritization approach may put more emphasis on things that have low probability but huge impact—or, vice versa, may put a lot of priority on things that are very likely and, hopefully, have a reasonable impact. You can introduce that along the way. When you come back to the synthesis, you just need to be nuanced about what you’re understanding, the likelihood.

Often, people lack humility in the way they make their recommendations: “This is the answer.” They’re very precise, and I think we would all be well-served to say, “This is a likely answer under the following sets of conditions” and then make the level of uncertainty clearer, if that is appropriate. It doesn’t mean you’re always in the gray zone; it doesn’t mean you don’t have a point of view. It just means that you can be explicit about the certainty of your answer when you make that recommendation.

Simon London: So it sounds like there is an underlying principle: “Acknowledge and embrace the uncertainty. Don’t pretend that it isn’t there. Be very clear about what the uncertainties are up front, and then build that into every step of the process.”

Hugo Sarrazin: Every step of the process.

Simon London: Yeah. We have just walked through a particular structured methodology for problem solving. But, of course, this is not the only structured methodology for problem solving. One that is also very well-known is design thinking, which comes at things very differently. So, Hugo, I know you have worked with a lot of designers. Just give us a very quick summary. Design thinking—what is it, and how does it relate?

Hugo Sarrazin: It starts with an incredible amount of empathy for the user and uses that to define the problem. It does pause and go out in the wild and spend an enormous amount of time seeing how people interact with objects, seeing the experience they’re getting, seeing the pain points or joy—and uses that to infer and define the problem.

Simon London: Problem definition, but out in the world.

Hugo Sarrazin: With an enormous amount of empathy. There’s a huge emphasis on empathy. Traditional, more classic problem solving is you define the problem based on an understanding of the situation. This one almost presupposes that we don’t know the problem until we go see it. The second thing is you need to come up with multiple scenarios or answers or ideas or concepts, and there’s a lot of divergent thinking initially. That’s slightly different, versus the prioritization, but not for long. Eventually, you need to kind of say, “OK, I’m going to converge again.” Then you go and you bring things back to the customer and get feedback and iterate. Then you rinse and repeat, rinse and repeat. There’s a lot of tactile building, along the way, of prototypes and things like that. It’s very iterative.

Simon London: So, Charles, are these complements or are these alternatives?

Charles Conn: I think they’re entirely complementary, and I think Hugo’s description is perfect. When we do problem definition well in classic problem solving, we are demonstrating the kind of empathy, at the very beginning of our problem, that design thinking asks us to approach. When we ideate—and that’s very similar to the disaggregation, prioritization, and work-planning steps—we do precisely the same thing, and often we use contrasting teams, so that we do have divergent thinking. The best teams allow divergent thinking to bump them off whatever their initial biases in problem solving are. For me, design thinking gives us a constant reminder of creativity, empathy, and the tactile nature of problem solving, but it’s absolutely complementary, not alternative.

Simon London: I think, in a world of cross-functional teams, an interesting question is do people with design-thinking backgrounds really work well together with classical problem solvers? How do you make that chemistry happen?

Hugo Sarrazin: Yeah, it is not easy when people have spent an enormous amount of time seeped in design thinking or user-centric design, whichever word you want to use. If the person who’s applying classic problem-solving methodology is very rigid and mechanical in the way they’re doing it, there could be an enormous amount of tension. If there’s not clarity in the role and not clarity in the process, I think having the two together can be, sometimes, problematic.

The second thing that happens often is that the artifacts the two methodologies try to gravitate toward can be different. Classic problem solving often gravitates toward a model; design thinking migrates toward a prototype. Rather than writing a big deck with all my supporting evidence, they’ll bring an example, a thing, and that feels different. Then you spend your time differently to achieve those two end products, so that’s another source of friction.

Now, I still think it can be an incredibly powerful thing to have the two—if there are the right people with the right mind-set, if there is a team that is explicit about the roles, if we’re clear about the kind of outcomes we are attempting to bring forward. There’s an enormous amount of collaborativeness and respect.

Simon London: But they have to respect each other’s methodology and be prepared to flex, maybe, a little bit, in how this process is going to work.

Hugo Sarrazin: Absolutely.

Simon London: The other area where, it strikes me, there could be a little bit of a different sort of friction is this whole concept of the day-one answer, which is what we were just talking about in classical problem solving. Now, you know that this is probably not going to be your final answer, but that’s how you begin to structure the problem. Whereas I would imagine your design thinkers—no, they’re going off to do their ethnographic research and get out into the field, potentially for a long time, before they come back with at least an initial hypothesis.

Want better strategies? Become a bulletproof problem solver

Hugo Sarrazin: That is a great callout, and that’s another difference. Designers typically will like to soak into the situation and avoid converging too quickly. There’s optionality and exploring different options. There’s a strong belief that keeps the solution space wide enough that you can come up with more radical ideas. If there’s a large design team or many designers on the team, and you come on Friday and say, “What’s our week-one answer?” they’re going to struggle. They’re not going to be comfortable, naturally, to give that answer. It doesn’t mean they don’t have an answer; it’s just not where they are in their thinking process.

Simon London: I think we are, sadly, out of time for today. But Charles and Hugo, thank you so much.

Charles Conn: It was a pleasure to be here, Simon.

Hugo Sarrazin: It was a pleasure. Thank you.

Simon London: And thanks, as always, to you, our listeners, for tuning into this episode of the McKinsey Podcast . If you want to learn more about problem solving, you can find the book, Bulletproof Problem Solving: The One Skill That Changes Everything , online or order it through your local bookstore. To learn more about McKinsey, you can of course find us at McKinsey.com.

Charles Conn is CEO of Oxford Sciences Innovation and an alumnus of McKinsey’s Sydney office. Hugo Sarrazin is a senior partner in the Silicon Valley office, where Simon London, a member of McKinsey Publishing, is also based.

Explore a career with us

Related articles.

Strategy to beat the odds

Five routes to more innovative problem solving

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 11 January 2023

The effectiveness of collaborative problem solving in promoting students’ critical thinking: A meta-analysis based on empirical literature

• Enwei Xu   ORCID: orcid.org/0000-0001-6424-8169 1 ,
• Wei Wang 1 &
• Qingxia Wang 1  

Humanities and Social Sciences Communications volume  10 , Article number:  16 ( 2023 ) Cite this article

14k Accesses

13 Citations

3 Altmetric

Metrics details

• Science, technology and society

Collaborative problem-solving has been widely embraced in the classroom instruction of critical thinking, which is regarded as the core of curriculum reform based on key competencies in the field of education as well as a key competence for learners in the 21st century. However, the effectiveness of collaborative problem-solving in promoting students’ critical thinking remains uncertain. This current research presents the major findings of a meta-analysis of 36 pieces of the literature revealed in worldwide educational periodicals during the 21st century to identify the effectiveness of collaborative problem-solving in promoting students’ critical thinking and to determine, based on evidence, whether and to what extent collaborative problem solving can result in a rise or decrease in critical thinking. The findings show that (1) collaborative problem solving is an effective teaching approach to foster students’ critical thinking, with a significant overall effect size (ES = 0.82, z  = 12.78, P  < 0.01, 95% CI [0.69, 0.95]); (2) in respect to the dimensions of critical thinking, collaborative problem solving can significantly and successfully enhance students’ attitudinal tendencies (ES = 1.17, z  = 7.62, P  < 0.01, 95% CI[0.87, 1.47]); nevertheless, it falls short in terms of improving students’ cognitive skills, having only an upper-middle impact (ES = 0.70, z  = 11.55, P  < 0.01, 95% CI[0.58, 0.82]); and (3) the teaching type (chi 2  = 7.20, P  < 0.05), intervention duration (chi 2  = 12.18, P  < 0.01), subject area (chi 2  = 13.36, P  < 0.05), group size (chi 2  = 8.77, P  < 0.05), and learning scaffold (chi 2  = 9.03, P  < 0.01) all have an impact on critical thinking, and they can be viewed as important moderating factors that affect how critical thinking develops. On the basis of these results, recommendations are made for further study and instruction to better support students’ critical thinking in the context of collaborative problem-solving.

Similar content being viewed by others

Impact of artificial intelligence on human loss in decision making, laziness and safety in education

A systematic review and multivariate meta-analysis of the physical and mental health benefits of touch interventions

Artificial intelligence and illusions of understanding in scientific research

Introduction.

Although critical thinking has a long history in research, the concept of critical thinking, which is regarded as an essential competence for learners in the 21st century, has recently attracted more attention from researchers and teaching practitioners (National Research Council, 2012 ). Critical thinking should be the core of curriculum reform based on key competencies in the field of education (Peng and Deng, 2017 ) because students with critical thinking can not only understand the meaning of knowledge but also effectively solve practical problems in real life even after knowledge is forgotten (Kek and Huijser, 2011 ). The definition of critical thinking is not universal (Ennis, 1989 ; Castle, 2009 ; Niu et al., 2013 ). In general, the definition of critical thinking is a self-aware and self-regulated thought process (Facione, 1990 ; Niu et al., 2013 ). It refers to the cognitive skills needed to interpret, analyze, synthesize, reason, and evaluate information as well as the attitudinal tendency to apply these abilities (Halpern, 2001 ). The view that critical thinking can be taught and learned through curriculum teaching has been widely supported by many researchers (e.g., Kuncel, 2011 ; Leng and Lu, 2020 ), leading to educators’ efforts to foster it among students. In the field of teaching practice, there are three types of courses for teaching critical thinking (Ennis, 1989 ). The first is an independent curriculum in which critical thinking is taught and cultivated without involving the knowledge of specific disciplines; the second is an integrated curriculum in which critical thinking is integrated into the teaching of other disciplines as a clear teaching goal; and the third is a mixed curriculum in which critical thinking is taught in parallel to the teaching of other disciplines for mixed teaching training. Furthermore, numerous measuring tools have been developed by researchers and educators to measure critical thinking in the context of teaching practice. These include standardized measurement tools, such as WGCTA, CCTST, CCTT, and CCTDI, which have been verified by repeated experiments and are considered effective and reliable by international scholars (Facione and Facione, 1992 ). In short, descriptions of critical thinking, including its two dimensions of attitudinal tendency and cognitive skills, different types of teaching courses, and standardized measurement tools provide a complex normative framework for understanding, teaching, and evaluating critical thinking.

Cultivating critical thinking in curriculum teaching can start with a problem, and one of the most popular critical thinking instructional approaches is problem-based learning (Liu et al., 2020 ). Duch et al. ( 2001 ) noted that problem-based learning in group collaboration is progressive active learning, which can improve students’ critical thinking and problem-solving skills. Collaborative problem-solving is the organic integration of collaborative learning and problem-based learning, which takes learners as the center of the learning process and uses problems with poor structure in real-world situations as the starting point for the learning process (Liang et al., 2017 ). Students learn the knowledge needed to solve problems in a collaborative group, reach a consensus on problems in the field, and form solutions through social cooperation methods, such as dialogue, interpretation, questioning, debate, negotiation, and reflection, thus promoting the development of learners’ domain knowledge and critical thinking (Cindy, 2004 ; Liang et al., 2017 ).

Collaborative problem-solving has been widely used in the teaching practice of critical thinking, and several studies have attempted to conduct a systematic review and meta-analysis of the empirical literature on critical thinking from various perspectives. However, little attention has been paid to the impact of collaborative problem-solving on critical thinking. Therefore, the best approach for developing and enhancing critical thinking throughout collaborative problem-solving is to examine how to implement critical thinking instruction; however, this issue is still unexplored, which means that many teachers are incapable of better instructing critical thinking (Leng and Lu, 2020 ; Niu et al., 2013 ). For example, Huber ( 2016 ) provided the meta-analysis findings of 71 publications on gaining critical thinking over various time frames in college with the aim of determining whether critical thinking was truly teachable. These authors found that learners significantly improve their critical thinking while in college and that critical thinking differs with factors such as teaching strategies, intervention duration, subject area, and teaching type. The usefulness of collaborative problem-solving in fostering students’ critical thinking, however, was not determined by this study, nor did it reveal whether there existed significant variations among the different elements. A meta-analysis of 31 pieces of educational literature was conducted by Liu et al. ( 2020 ) to assess the impact of problem-solving on college students’ critical thinking. These authors found that problem-solving could promote the development of critical thinking among college students and proposed establishing a reasonable group structure for problem-solving in a follow-up study to improve students’ critical thinking. Additionally, previous empirical studies have reached inconclusive and even contradictory conclusions about whether and to what extent collaborative problem-solving increases or decreases critical thinking levels. As an illustration, Yang et al. ( 2008 ) carried out an experiment on the integrated curriculum teaching of college students based on a web bulletin board with the goal of fostering participants’ critical thinking in the context of collaborative problem-solving. These authors’ research revealed that through sharing, debating, examining, and reflecting on various experiences and ideas, collaborative problem-solving can considerably enhance students’ critical thinking in real-life problem situations. In contrast, collaborative problem-solving had a positive impact on learners’ interaction and could improve learning interest and motivation but could not significantly improve students’ critical thinking when compared to traditional classroom teaching, according to research by Naber and Wyatt ( 2014 ) and Sendag and Odabasi ( 2009 ) on undergraduate and high school students, respectively.

The above studies show that there is inconsistency regarding the effectiveness of collaborative problem-solving in promoting students’ critical thinking. Therefore, it is essential to conduct a thorough and trustworthy review to detect and decide whether and to what degree collaborative problem-solving can result in a rise or decrease in critical thinking. Meta-analysis is a quantitative analysis approach that is utilized to examine quantitative data from various separate studies that are all focused on the same research topic. This approach characterizes the effectiveness of its impact by averaging the effect sizes of numerous qualitative studies in an effort to reduce the uncertainty brought on by independent research and produce more conclusive findings (Lipsey and Wilson, 2001 ).

This paper used a meta-analytic approach and carried out a meta-analysis to examine the effectiveness of collaborative problem-solving in promoting students’ critical thinking in order to make a contribution to both research and practice. The following research questions were addressed by this meta-analysis:

What is the overall effect size of collaborative problem-solving in promoting students’ critical thinking and its impact on the two dimensions of critical thinking (i.e., attitudinal tendency and cognitive skills)?

How are the disparities between the study conclusions impacted by various moderating variables if the impacts of various experimental designs in the included studies are heterogeneous?

This research followed the strict procedures (e.g., database searching, identification, screening, eligibility, merging, duplicate removal, and analysis of included studies) of Cooper’s ( 2010 ) proposed meta-analysis approach for examining quantitative data from various separate studies that are all focused on the same research topic. The relevant empirical research that appeared in worldwide educational periodicals within the 21st century was subjected to this meta-analysis using Rev-Man 5.4. The consistency of the data extracted separately by two researchers was tested using Cohen’s kappa coefficient, and a publication bias test and a heterogeneity test were run on the sample data to ascertain the quality of this meta-analysis.

Data sources and search strategies

There were three stages to the data collection process for this meta-analysis, as shown in Fig. 1 , which shows the number of articles included and eliminated during the selection process based on the statement and study eligibility criteria.

This flowchart shows the number of records identified, included and excluded in the article.

First, the databases used to systematically search for relevant articles were the journal papers of the Web of Science Core Collection and the Chinese Core source journal, as well as the Chinese Social Science Citation Index (CSSCI) source journal papers included in CNKI. These databases were selected because they are credible platforms that are sources of scholarly and peer-reviewed information with advanced search tools and contain literature relevant to the subject of our topic from reliable researchers and experts. The search string with the Boolean operator used in the Web of Science was “TS = (((“critical thinking” or “ct” and “pretest” or “posttest”) or (“critical thinking” or “ct” and “control group” or “quasi experiment” or “experiment”)) and (“collaboration” or “collaborative learning” or “CSCL”) and (“problem solving” or “problem-based learning” or “PBL”))”. The research area was “Education Educational Research”, and the search period was “January 1, 2000, to December 30, 2021”. A total of 412 papers were obtained. The search string with the Boolean operator used in the CNKI was “SU = (‘critical thinking’*‘collaboration’ + ‘critical thinking’*‘collaborative learning’ + ‘critical thinking’*‘CSCL’ + ‘critical thinking’*‘problem solving’ + ‘critical thinking’*‘problem-based learning’ + ‘critical thinking’*‘PBL’ + ‘critical thinking’*‘problem oriented’) AND FT = (‘experiment’ + ‘quasi experiment’ + ‘pretest’ + ‘posttest’ + ‘empirical study’)” (translated into Chinese when searching). A total of 56 studies were found throughout the search period of “January 2000 to December 2021”. From the databases, all duplicates and retractions were eliminated before exporting the references into Endnote, a program for managing bibliographic references. In all, 466 studies were found.

Second, the studies that matched the inclusion and exclusion criteria for the meta-analysis were chosen by two researchers after they had reviewed the abstracts and titles of the gathered articles, yielding a total of 126 studies.

Third, two researchers thoroughly reviewed each included article’s whole text in accordance with the inclusion and exclusion criteria. Meanwhile, a snowball search was performed using the references and citations of the included articles to ensure complete coverage of the articles. Ultimately, 36 articles were kept.

Two researchers worked together to carry out this entire process, and a consensus rate of almost 94.7% was reached after discussion and negotiation to clarify any emerging differences.

Eligibility criteria

Since not all the retrieved studies matched the criteria for this meta-analysis, eligibility criteria for both inclusion and exclusion were developed as follows:

The publication language of the included studies was limited to English and Chinese, and the full text could be obtained. Articles that did not meet the publication language and articles not published between 2000 and 2021 were excluded.

The research design of the included studies must be empirical and quantitative studies that can assess the effect of collaborative problem-solving on the development of critical thinking. Articles that could not identify the causal mechanisms by which collaborative problem-solving affects critical thinking, such as review articles and theoretical articles, were excluded.

The research method of the included studies must feature a randomized control experiment or a quasi-experiment, or a natural experiment, which have a higher degree of internal validity with strong experimental designs and can all plausibly provide evidence that critical thinking and collaborative problem-solving are causally related. Articles with non-experimental research methods, such as purely correlational or observational studies, were excluded.

The participants of the included studies were only students in school, including K-12 students and college students. Articles in which the participants were non-school students, such as social workers or adult learners, were excluded.

The research results of the included studies must mention definite signs that may be utilized to gauge critical thinking’s impact (e.g., sample size, mean value, or standard deviation). Articles that lacked specific measurement indicators for critical thinking and could not calculate the effect size were excluded.

Data coding design

In order to perform a meta-analysis, it is necessary to collect the most important information from the articles, codify that information’s properties, and convert descriptive data into quantitative data. Therefore, this study designed a data coding template (see Table 1 ). Ultimately, 16 coding fields were retained.

The designed data-coding template consisted of three pieces of information. Basic information about the papers was included in the descriptive information: the publishing year, author, serial number, and title of the paper.

The variable information for the experimental design had three variables: the independent variable (instruction method), the dependent variable (critical thinking), and the moderating variable (learning stage, teaching type, intervention duration, learning scaffold, group size, measuring tool, and subject area). Depending on the topic of this study, the intervention strategy, as the independent variable, was coded into collaborative and non-collaborative problem-solving. The dependent variable, critical thinking, was coded as a cognitive skill and an attitudinal tendency. And seven moderating variables were created by grouping and combining the experimental design variables discovered within the 36 studies (see Table 1 ), where learning stages were encoded as higher education, high school, middle school, and primary school or lower; teaching types were encoded as mixed courses, integrated courses, and independent courses; intervention durations were encoded as 0–1 weeks, 1–4 weeks, 4–12 weeks, and more than 12 weeks; group sizes were encoded as 2–3 persons, 4–6 persons, 7–10 persons, and more than 10 persons; learning scaffolds were encoded as teacher-supported learning scaffold, technique-supported learning scaffold, and resource-supported learning scaffold; measuring tools were encoded as standardized measurement tools (e.g., WGCTA, CCTT, CCTST, and CCTDI) and self-adapting measurement tools (e.g., modified or made by researchers); and subject areas were encoded according to the specific subjects used in the 36 included studies.

The data information contained three metrics for measuring critical thinking: sample size, average value, and standard deviation. It is vital to remember that studies with various experimental designs frequently adopt various formulas to determine the effect size. And this paper used Morris’ proposed standardized mean difference (SMD) calculation formula ( 2008 , p. 369; see Supplementary Table S3 ).

Procedure for extracting and coding data

According to the data coding template (see Table 1 ), the 36 papers’ information was retrieved by two researchers, who then entered them into Excel (see Supplementary Table S1 ). The results of each study were extracted separately in the data extraction procedure if an article contained numerous studies on critical thinking, or if a study assessed different critical thinking dimensions. For instance, Tiwari et al. ( 2010 ) used four time points, which were viewed as numerous different studies, to examine the outcomes of critical thinking, and Chen ( 2013 ) included the two outcome variables of attitudinal tendency and cognitive skills, which were regarded as two studies. After discussion and negotiation during data extraction, the two researchers’ consistency test coefficients were roughly 93.27%. Supplementary Table S2 details the key characteristics of the 36 included articles with 79 effect quantities, including descriptive information (e.g., the publishing year, author, serial number, and title of the paper), variable information (e.g., independent variables, dependent variables, and moderating variables), and data information (e.g., mean values, standard deviations, and sample size). Following that, testing for publication bias and heterogeneity was done on the sample data using the Rev-Man 5.4 software, and then the test results were used to conduct a meta-analysis.

Publication bias test

When the sample of studies included in a meta-analysis does not accurately reflect the general status of research on the relevant subject, publication bias is said to be exhibited in this research. The reliability and accuracy of the meta-analysis may be impacted by publication bias. Due to this, the meta-analysis needs to check the sample data for publication bias (Stewart et al., 2006 ). A popular method to check for publication bias is the funnel plot; and it is unlikely that there will be publishing bias when the data are equally dispersed on either side of the average effect size and targeted within the higher region. The data are equally dispersed within the higher portion of the efficient zone, consistent with the funnel plot connected with this analysis (see Fig. 2 ), indicating that publication bias is unlikely in this situation.

This funnel plot shows the result of publication bias of 79 effect quantities across 36 studies.

Heterogeneity test

To select the appropriate effect models for the meta-analysis, one might use the results of a heterogeneity test on the data effect sizes. In a meta-analysis, it is common practice to gauge the degree of data heterogeneity using the I 2 value, and I 2  ≥ 50% is typically understood to denote medium-high heterogeneity, which calls for the adoption of a random effect model; if not, a fixed effect model ought to be applied (Lipsey and Wilson, 2001 ). The findings of the heterogeneity test in this paper (see Table 2 ) revealed that I 2 was 86% and displayed significant heterogeneity ( P  < 0.01). To ensure accuracy and reliability, the overall effect size ought to be calculated utilizing the random effect model.

The analysis of the overall effect size

This meta-analysis utilized a random effect model to examine 79 effect quantities from 36 studies after eliminating heterogeneity. In accordance with Cohen’s criterion (Cohen, 1992 ), it is abundantly clear from the analysis results, which are shown in the forest plot of the overall effect (see Fig. 3 ), that the cumulative impact size of cooperative problem-solving is 0.82, which is statistically significant ( z  = 12.78, P  < 0.01, 95% CI [0.69, 0.95]), and can encourage learners to practice critical thinking.

This forest plot shows the analysis result of the overall effect size across 36 studies.

In addition, this study examined two distinct dimensions of critical thinking to better understand the precise contributions that collaborative problem-solving makes to the growth of critical thinking. The findings (see Table 3 ) indicate that collaborative problem-solving improves cognitive skills (ES = 0.70) and attitudinal tendency (ES = 1.17), with significant intergroup differences (chi 2  = 7.95, P  < 0.01). Although collaborative problem-solving improves both dimensions of critical thinking, it is essential to point out that the improvements in students’ attitudinal tendency are much more pronounced and have a significant comprehensive effect (ES = 1.17, z  = 7.62, P  < 0.01, 95% CI [0.87, 1.47]), whereas gains in learners’ cognitive skill are slightly improved and are just above average. (ES = 0.70, z  = 11.55, P  < 0.01, 95% CI [0.58, 0.82]).

The analysis of moderator effect size

The whole forest plot’s 79 effect quantities underwent a two-tailed test, which revealed significant heterogeneity ( I 2  = 86%, z  = 12.78, P  < 0.01), indicating differences between various effect sizes that may have been influenced by moderating factors other than sampling error. Therefore, exploring possible moderating factors that might produce considerable heterogeneity was done using subgroup analysis, such as the learning stage, learning scaffold, teaching type, group size, duration of the intervention, measuring tool, and the subject area included in the 36 experimental designs, in order to further explore the key factors that influence critical thinking. The findings (see Table 4 ) indicate that various moderating factors have advantageous effects on critical thinking. In this situation, the subject area (chi 2  = 13.36, P  < 0.05), group size (chi 2  = 8.77, P  < 0.05), intervention duration (chi 2  = 12.18, P  < 0.01), learning scaffold (chi 2  = 9.03, P  < 0.01), and teaching type (chi 2  = 7.20, P  < 0.05) are all significant moderators that can be applied to support the cultivation of critical thinking. However, since the learning stage and the measuring tools did not significantly differ among intergroup (chi 2  = 3.15, P  = 0.21 > 0.05, and chi 2  = 0.08, P  = 0.78 > 0.05), we are unable to explain why these two factors are crucial in supporting the cultivation of critical thinking in the context of collaborative problem-solving. These are the precise outcomes, as follows:

Various learning stages influenced critical thinking positively, without significant intergroup differences (chi 2  = 3.15, P  = 0.21 > 0.05). High school was first on the list of effect sizes (ES = 1.36, P  < 0.01), then higher education (ES = 0.78, P  < 0.01), and middle school (ES = 0.73, P  < 0.01). These results show that, despite the learning stage’s beneficial influence on cultivating learners’ critical thinking, we are unable to explain why it is essential for cultivating critical thinking in the context of collaborative problem-solving.

Different teaching types had varying degrees of positive impact on critical thinking, with significant intergroup differences (chi 2  = 7.20, P  < 0.05). The effect size was ranked as follows: mixed courses (ES = 1.34, P  < 0.01), integrated courses (ES = 0.81, P  < 0.01), and independent courses (ES = 0.27, P  < 0.01). These results indicate that the most effective approach to cultivate critical thinking utilizing collaborative problem solving is through the teaching type of mixed courses.

Various intervention durations significantly improved critical thinking, and there were significant intergroup differences (chi 2  = 12.18, P  < 0.01). The effect sizes related to this variable showed a tendency to increase with longer intervention durations. The improvement in critical thinking reached a significant level (ES = 0.85, P  < 0.01) after more than 12 weeks of training. These findings indicate that the intervention duration and critical thinking’s impact are positively correlated, with a longer intervention duration having a greater effect.

Different learning scaffolds influenced critical thinking positively, with significant intergroup differences (chi 2  = 9.03, P  < 0.01). The resource-supported learning scaffold (ES = 0.69, P  < 0.01) acquired a medium-to-higher level of impact, the technique-supported learning scaffold (ES = 0.63, P  < 0.01) also attained a medium-to-higher level of impact, and the teacher-supported learning scaffold (ES = 0.92, P  < 0.01) displayed a high level of significant impact. These results show that the learning scaffold with teacher support has the greatest impact on cultivating critical thinking.

Various group sizes influenced critical thinking positively, and the intergroup differences were statistically significant (chi 2  = 8.77, P  < 0.05). Critical thinking showed a general declining trend with increasing group size. The overall effect size of 2–3 people in this situation was the biggest (ES = 0.99, P  < 0.01), and when the group size was greater than 7 people, the improvement in critical thinking was at the lower-middle level (ES < 0.5, P  < 0.01). These results show that the impact on critical thinking is positively connected with group size, and as group size grows, so does the overall impact.

Various measuring tools influenced critical thinking positively, with significant intergroup differences (chi 2  = 0.08, P  = 0.78 > 0.05). In this situation, the self-adapting measurement tools obtained an upper-medium level of effect (ES = 0.78), whereas the complete effect size of the standardized measurement tools was the largest, achieving a significant level of effect (ES = 0.84, P  < 0.01). These results show that, despite the beneficial influence of the measuring tool on cultivating critical thinking, we are unable to explain why it is crucial in fostering the growth of critical thinking by utilizing the approach of collaborative problem-solving.

Different subject areas had a greater impact on critical thinking, and the intergroup differences were statistically significant (chi 2  = 13.36, P  < 0.05). Mathematics had the greatest overall impact, achieving a significant level of effect (ES = 1.68, P  < 0.01), followed by science (ES = 1.25, P  < 0.01) and medical science (ES = 0.87, P  < 0.01), both of which also achieved a significant level of effect. Programming technology was the least effective (ES = 0.39, P  < 0.01), only having a medium-low degree of effect compared to education (ES = 0.72, P  < 0.01) and other fields (such as language, art, and social sciences) (ES = 0.58, P  < 0.01). These results suggest that scientific fields (e.g., mathematics, science) may be the most effective subject areas for cultivating critical thinking utilizing the approach of collaborative problem-solving.

The effectiveness of collaborative problem solving with regard to teaching critical thinking

According to this meta-analysis, using collaborative problem-solving as an intervention strategy in critical thinking teaching has a considerable amount of impact on cultivating learners’ critical thinking as a whole and has a favorable promotional effect on the two dimensions of critical thinking. According to certain studies, collaborative problem solving, the most frequently used critical thinking teaching strategy in curriculum instruction can considerably enhance students’ critical thinking (e.g., Liang et al., 2017 ; Liu et al., 2020 ; Cindy, 2004 ). This meta-analysis provides convergent data support for the above research views. Thus, the findings of this meta-analysis not only effectively address the first research query regarding the overall effect of cultivating critical thinking and its impact on the two dimensions of critical thinking (i.e., attitudinal tendency and cognitive skills) utilizing the approach of collaborative problem-solving, but also enhance our confidence in cultivating critical thinking by using collaborative problem-solving intervention approach in the context of classroom teaching.

Furthermore, the associated improvements in attitudinal tendency are much stronger, but the corresponding improvements in cognitive skill are only marginally better. According to certain studies, cognitive skill differs from the attitudinal tendency in classroom instruction; the cultivation and development of the former as a key ability is a process of gradual accumulation, while the latter as an attitude is affected by the context of the teaching situation (e.g., a novel and exciting teaching approach, challenging and rewarding tasks) (Halpern, 2001 ; Wei and Hong, 2022 ). Collaborative problem-solving as a teaching approach is exciting and interesting, as well as rewarding and challenging; because it takes the learners as the focus and examines problems with poor structure in real situations, and it can inspire students to fully realize their potential for problem-solving, which will significantly improve their attitudinal tendency toward solving problems (Liu et al., 2020 ). Similar to how collaborative problem-solving influences attitudinal tendency, attitudinal tendency impacts cognitive skill when attempting to solve a problem (Liu et al., 2020 ; Zhang et al., 2022 ), and stronger attitudinal tendencies are associated with improved learning achievement and cognitive ability in students (Sison, 2008 ; Zhang et al., 2022 ). It can be seen that the two specific dimensions of critical thinking as well as critical thinking as a whole are affected by collaborative problem-solving, and this study illuminates the nuanced links between cognitive skills and attitudinal tendencies with regard to these two dimensions of critical thinking. To fully develop students’ capacity for critical thinking, future empirical research should pay closer attention to cognitive skills.

The moderating effects of collaborative problem solving with regard to teaching critical thinking

In order to further explore the key factors that influence critical thinking, exploring possible moderating effects that might produce considerable heterogeneity was done using subgroup analysis. The findings show that the moderating factors, such as the teaching type, learning stage, group size, learning scaffold, duration of the intervention, measuring tool, and the subject area included in the 36 experimental designs, could all support the cultivation of collaborative problem-solving in critical thinking. Among them, the effect size differences between the learning stage and measuring tool are not significant, which does not explain why these two factors are crucial in supporting the cultivation of critical thinking utilizing the approach of collaborative problem-solving.

In terms of the learning stage, various learning stages influenced critical thinking positively without significant intergroup differences, indicating that we are unable to explain why it is crucial in fostering the growth of critical thinking.

Although high education accounts for 70.89% of all empirical studies performed by researchers, high school may be the appropriate learning stage to foster students’ critical thinking by utilizing the approach of collaborative problem-solving since it has the largest overall effect size. This phenomenon may be related to student’s cognitive development, which needs to be further studied in follow-up research.

With regard to teaching type, mixed course teaching may be the best teaching method to cultivate students’ critical thinking. Relevant studies have shown that in the actual teaching process if students are trained in thinking methods alone, the methods they learn are isolated and divorced from subject knowledge, which is not conducive to their transfer of thinking methods; therefore, if students’ thinking is trained only in subject teaching without systematic method training, it is challenging to apply to real-world circumstances (Ruggiero, 2012 ; Hu and Liu, 2015 ). Teaching critical thinking as mixed course teaching in parallel to other subject teachings can achieve the best effect on learners’ critical thinking, and explicit critical thinking instruction is more effective than less explicit critical thinking instruction (Bensley and Spero, 2014 ).

In terms of the intervention duration, with longer intervention times, the overall effect size shows an upward tendency. Thus, the intervention duration and critical thinking’s impact are positively correlated. Critical thinking, as a key competency for students in the 21st century, is difficult to get a meaningful improvement in a brief intervention duration. Instead, it could be developed over a lengthy period of time through consistent teaching and the progressive accumulation of knowledge (Halpern, 2001 ; Hu and Liu, 2015 ). Therefore, future empirical studies ought to take these restrictions into account throughout a longer period of critical thinking instruction.

With regard to group size, a group size of 2–3 persons has the highest effect size, and the comprehensive effect size decreases with increasing group size in general. This outcome is in line with some research findings; as an example, a group composed of two to four members is most appropriate for collaborative learning (Schellens and Valcke, 2006 ). However, the meta-analysis results also indicate that once the group size exceeds 7 people, small groups cannot produce better interaction and performance than large groups. This may be because the learning scaffolds of technique support, resource support, and teacher support improve the frequency and effectiveness of interaction among group members, and a collaborative group with more members may increase the diversity of views, which is helpful to cultivate critical thinking utilizing the approach of collaborative problem-solving.

With regard to the learning scaffold, the three different kinds of learning scaffolds can all enhance critical thinking. Among them, the teacher-supported learning scaffold has the largest overall effect size, demonstrating the interdependence of effective learning scaffolds and collaborative problem-solving. This outcome is in line with some research findings; as an example, a successful strategy is to encourage learners to collaborate, come up with solutions, and develop critical thinking skills by using learning scaffolds (Reiser, 2004 ; Xu et al., 2022 ); learning scaffolds can lower task complexity and unpleasant feelings while also enticing students to engage in learning activities (Wood et al., 2006 ); learning scaffolds are designed to assist students in using learning approaches more successfully to adapt the collaborative problem-solving process, and the teacher-supported learning scaffolds have the greatest influence on critical thinking in this process because they are more targeted, informative, and timely (Xu et al., 2022 ).

With respect to the measuring tool, despite the fact that standardized measurement tools (such as the WGCTA, CCTT, and CCTST) have been acknowledged as trustworthy and effective by worldwide experts, only 54.43% of the research included in this meta-analysis adopted them for assessment, and the results indicated no intergroup differences. These results suggest that not all teaching circumstances are appropriate for measuring critical thinking using standardized measurement tools. “The measuring tools for measuring thinking ability have limits in assessing learners in educational situations and should be adapted appropriately to accurately assess the changes in learners’ critical thinking.”, according to Simpson and Courtney ( 2002 , p. 91). As a result, in order to more fully and precisely gauge how learners’ critical thinking has evolved, we must properly modify standardized measuring tools based on collaborative problem-solving learning contexts.

With regard to the subject area, the comprehensive effect size of science departments (e.g., mathematics, science, medical science) is larger than that of language arts and social sciences. Some recent international education reforms have noted that critical thinking is a basic part of scientific literacy. Students with scientific literacy can prove the rationality of their judgment according to accurate evidence and reasonable standards when they face challenges or poorly structured problems (Kyndt et al., 2013 ), which makes critical thinking crucial for developing scientific understanding and applying this understanding to practical problem solving for problems related to science, technology, and society (Yore et al., 2007 ).

Suggestions for critical thinking teaching

Other than those stated in the discussion above, the following suggestions are offered for critical thinking instruction utilizing the approach of collaborative problem-solving.

First, teachers should put a special emphasis on the two core elements, which are collaboration and problem-solving, to design real problems based on collaborative situations. This meta-analysis provides evidence to support the view that collaborative problem-solving has a strong synergistic effect on promoting students’ critical thinking. Asking questions about real situations and allowing learners to take part in critical discussions on real problems during class instruction are key ways to teach critical thinking rather than simply reading speculative articles without practice (Mulnix, 2012 ). Furthermore, the improvement of students’ critical thinking is realized through cognitive conflict with other learners in the problem situation (Yang et al., 2008 ). Consequently, it is essential for teachers to put a special emphasis on the two core elements, which are collaboration and problem-solving, and design real problems and encourage students to discuss, negotiate, and argue based on collaborative problem-solving situations.

Second, teachers should design and implement mixed courses to cultivate learners’ critical thinking, utilizing the approach of collaborative problem-solving. Critical thinking can be taught through curriculum instruction (Kuncel, 2011 ; Leng and Lu, 2020 ), with the goal of cultivating learners’ critical thinking for flexible transfer and application in real problem-solving situations. This meta-analysis shows that mixed course teaching has a highly substantial impact on the cultivation and promotion of learners’ critical thinking. Therefore, teachers should design and implement mixed course teaching with real collaborative problem-solving situations in combination with the knowledge content of specific disciplines in conventional teaching, teach methods and strategies of critical thinking based on poorly structured problems to help students master critical thinking, and provide practical activities in which students can interact with each other to develop knowledge construction and critical thinking utilizing the approach of collaborative problem-solving.

Third, teachers should be more trained in critical thinking, particularly preservice teachers, and they also should be conscious of the ways in which teachers’ support for learning scaffolds can promote critical thinking. The learning scaffold supported by teachers had the greatest impact on learners’ critical thinking, in addition to being more directive, targeted, and timely (Wood et al., 2006 ). Critical thinking can only be effectively taught when teachers recognize the significance of critical thinking for students’ growth and use the proper approaches while designing instructional activities (Forawi, 2016 ). Therefore, with the intention of enabling teachers to create learning scaffolds to cultivate learners’ critical thinking utilizing the approach of collaborative problem solving, it is essential to concentrate on the teacher-supported learning scaffolds and enhance the instruction for teaching critical thinking to teachers, especially preservice teachers.

Implications and limitations

There are certain limitations in this meta-analysis, but future research can correct them. First, the search languages were restricted to English and Chinese, so it is possible that pertinent studies that were written in other languages were overlooked, resulting in an inadequate number of articles for review. Second, these data provided by the included studies are partially missing, such as whether teachers were trained in the theory and practice of critical thinking, the average age and gender of learners, and the differences in critical thinking among learners of various ages and genders. Third, as is typical for review articles, more studies were released while this meta-analysis was being done; therefore, it had a time limit. With the development of relevant research, future studies focusing on these issues are highly relevant and needed.

Conclusions

The subject of the magnitude of collaborative problem-solving’s impact on fostering students’ critical thinking, which received scant attention from other studies, was successfully addressed by this study. The question of the effectiveness of collaborative problem-solving in promoting students’ critical thinking was addressed in this study, which addressed a topic that had gotten little attention in earlier research. The following conclusions can be made:

Regarding the results obtained, collaborative problem solving is an effective teaching approach to foster learners’ critical thinking, with a significant overall effect size (ES = 0.82, z  = 12.78, P  < 0.01, 95% CI [0.69, 0.95]). With respect to the dimensions of critical thinking, collaborative problem-solving can significantly and effectively improve students’ attitudinal tendency, and the comprehensive effect is significant (ES = 1.17, z  = 7.62, P  < 0.01, 95% CI [0.87, 1.47]); nevertheless, it falls short in terms of improving students’ cognitive skills, having only an upper-middle impact (ES = 0.70, z  = 11.55, P  < 0.01, 95% CI [0.58, 0.82]).

As demonstrated by both the results and the discussion, there are varying degrees of beneficial effects on students’ critical thinking from all seven moderating factors, which were found across 36 studies. In this context, the teaching type (chi 2  = 7.20, P  < 0.05), intervention duration (chi 2  = 12.18, P  < 0.01), subject area (chi 2  = 13.36, P  < 0.05), group size (chi 2  = 8.77, P  < 0.05), and learning scaffold (chi 2  = 9.03, P  < 0.01) all have a positive impact on critical thinking, and they can be viewed as important moderating factors that affect how critical thinking develops. Since the learning stage (chi 2  = 3.15, P  = 0.21 > 0.05) and measuring tools (chi 2  = 0.08, P  = 0.78 > 0.05) did not demonstrate any significant intergroup differences, we are unable to explain why these two factors are crucial in supporting the cultivation of critical thinking in the context of collaborative problem-solving.

Data availability

All data generated or analyzed during this study are included within the article and its supplementary information files, and the supplementary information files are available in the Dataverse repository: https://doi.org/10.7910/DVN/IPFJO6 .

Bensley DA, Spero RA (2014) Improving critical thinking skills and meta-cognitive monitoring through direct infusion. Think Skills Creat 12:55–68. https://doi.org/10.1016/j.tsc.2014.02.001

Article   Google Scholar  

Castle A (2009) Defining and assessing critical thinking skills for student radiographers. Radiography 15(1):70–76. https://doi.org/10.1016/j.radi.2007.10.007

Chen XD (2013) An empirical study on the influence of PBL teaching model on critical thinking ability of non-English majors. J PLA Foreign Lang College 36 (04):68–72

Google Scholar  

Cohen A (1992) Antecedents of organizational commitment across occupational groups: a meta-analysis. J Organ Behav. https://doi.org/10.1002/job.4030130602

Cooper H (2010) Research synthesis and meta-analysis: a step-by-step approach, 4th edn. Sage, London, England

Cindy HS (2004) Problem-based learning: what and how do students learn? Educ Psychol Rev 51(1):31–39

Duch BJ, Gron SD, Allen DE (2001) The power of problem-based learning: a practical “how to” for teaching undergraduate courses in any discipline. Stylus Educ Sci 2:190–198

Ennis RH (1989) Critical thinking and subject specificity: clarification and needed research. Educ Res 18(3):4–10. https://doi.org/10.3102/0013189x018003004

Facione PA (1990) Critical thinking: a statement of expert consensus for purposes of educational assessment and instruction. Research findings and recommendations. Eric document reproduction service. https://eric.ed.gov/?id=ed315423

Facione PA, Facione NC (1992) The California Critical Thinking Dispositions Inventory (CCTDI) and the CCTDI test manual. California Academic Press, Millbrae, CA

Forawi SA (2016) Standard-based science education and critical thinking. Think Skills Creat 20:52–62. https://doi.org/10.1016/j.tsc.2016.02.005

Halpern DF (2001) Assessing the effectiveness of critical thinking instruction. J Gen Educ 50(4):270–286. https://doi.org/10.2307/27797889

Hu WP, Liu J (2015) Cultivation of pupils’ thinking ability: a five-year follow-up study. Psychol Behav Res 13(05):648–654. https://doi.org/10.3969/j.issn.1672-0628.2015.05.010

Huber K (2016) Does college teach critical thinking? A meta-analysis. Rev Educ Res 86(2):431–468. https://doi.org/10.3102/0034654315605917

Kek MYCA, Huijser H (2011) The power of problem-based learning in developing critical thinking skills: preparing students for tomorrow’s digital futures in today’s classrooms. High Educ Res Dev 30(3):329–341. https://doi.org/10.1080/07294360.2010.501074

Kuncel NR (2011) Measurement and meaning of critical thinking (Research report for the NRC 21st Century Skills Workshop). National Research Council, Washington, DC

Kyndt E, Raes E, Lismont B, Timmers F, Cascallar E, Dochy F (2013) A meta-analysis of the effects of face-to-face cooperative learning. Do recent studies falsify or verify earlier findings? Educ Res Rev 10(2):133–149. https://doi.org/10.1016/j.edurev.2013.02.002

Leng J, Lu XX (2020) Is critical thinking really teachable?—A meta-analysis based on 79 experimental or quasi experimental studies. Open Educ Res 26(06):110–118. https://doi.org/10.13966/j.cnki.kfjyyj.2020.06.011

Liang YZ, Zhu K, Zhao CL (2017) An empirical study on the depth of interaction promoted by collaborative problem solving learning activities. J E-educ Res 38(10):87–92. https://doi.org/10.13811/j.cnki.eer.2017.10.014

Lipsey M, Wilson D (2001) Practical meta-analysis. International Educational and Professional, London, pp. 92–160

Liu Z, Wu W, Jiang Q (2020) A study on the influence of problem based learning on college students’ critical thinking-based on a meta-analysis of 31 studies. Explor High Educ 03:43–49

Morris SB (2008) Estimating effect sizes from pretest-posttest-control group designs. Organ Res Methods 11(2):364–386. https://doi.org/10.1177/1094428106291059

Article   ADS   Google Scholar  

Mulnix JW (2012) Thinking critically about critical thinking. Educ Philos Theory 44(5):464–479. https://doi.org/10.1111/j.1469-5812.2010.00673.x

Naber J, Wyatt TH (2014) The effect of reflective writing interventions on the critical thinking skills and dispositions of baccalaureate nursing students. Nurse Educ Today 34(1):67–72. https://doi.org/10.1016/j.nedt.2013.04.002

National Research Council (2012) Education for life and work: developing transferable knowledge and skills in the 21st century. The National Academies Press, Washington, DC

Niu L, Behar HLS, Garvan CW (2013) Do instructional interventions influence college students’ critical thinking skills? A meta-analysis. Educ Res Rev 9(12):114–128. https://doi.org/10.1016/j.edurev.2012.12.002

Peng ZM, Deng L (2017) Towards the core of education reform: cultivating critical thinking skills as the core of skills in the 21st century. Res Educ Dev 24:57–63. https://doi.org/10.14121/j.cnki.1008-3855.2017.24.011

Reiser BJ (2004) Scaffolding complex learning: the mechanisms of structuring and problematizing student work. J Learn Sci 13(3):273–304. https://doi.org/10.1207/s15327809jls1303_2

Ruggiero VR (2012) The art of thinking: a guide to critical and creative thought, 4th edn. Harper Collins College Publishers, New York

Schellens T, Valcke M (2006) Fostering knowledge construction in university students through asynchronous discussion groups. Comput Educ 46(4):349–370. https://doi.org/10.1016/j.compedu.2004.07.010

Sendag S, Odabasi HF (2009) Effects of an online problem based learning course on content knowledge acquisition and critical thinking skills. Comput Educ 53(1):132–141. https://doi.org/10.1016/j.compedu.2009.01.008

Sison R (2008) Investigating Pair Programming in a Software Engineering Course in an Asian Setting. 2008 15th Asia-Pacific Software Engineering Conference, pp. 325–331. https://doi.org/10.1109/APSEC.2008.61

Simpson E, Courtney M (2002) Critical thinking in nursing education: literature review. Mary Courtney 8(2):89–98

Stewart L, Tierney J, Burdett S (2006) Do systematic reviews based on individual patient data offer a means of circumventing biases associated with trial publications? Publication bias in meta-analysis. John Wiley and Sons Inc, New York, pp. 261–286

Tiwari A, Lai P, So M, Yuen K (2010) A comparison of the effects of problem-based learning and lecturing on the development of students’ critical thinking. Med Educ 40(6):547–554. https://doi.org/10.1111/j.1365-2929.2006.02481.x

Wood D, Bruner JS, Ross G (2006) The role of tutoring in problem solving. J Child Psychol Psychiatry 17(2):89–100. https://doi.org/10.1111/j.1469-7610.1976.tb00381.x

Wei T, Hong S (2022) The meaning and realization of teachable critical thinking. Educ Theory Practice 10:51–57

Xu EW, Wang W, Wang QX (2022) A meta-analysis of the effectiveness of programming teaching in promoting K-12 students’ computational thinking. Educ Inf Technol. https://doi.org/10.1007/s10639-022-11445-2

Yang YC, Newby T, Bill R (2008) Facilitating interactions through structured web-based bulletin boards: a quasi-experimental study on promoting learners’ critical thinking skills. Comput Educ 50(4):1572–1585. https://doi.org/10.1016/j.compedu.2007.04.006

Yore LD, Pimm D, Tuan HL (2007) The literacy component of mathematical and scientific literacy. Int J Sci Math Educ 5(4):559–589. https://doi.org/10.1007/s10763-007-9089-4

Zhang T, Zhang S, Gao QQ, Wang JH (2022) Research on the development of learners’ critical thinking in online peer review. Audio Visual Educ Res 6:53–60. https://doi.org/10.13811/j.cnki.eer.2022.06.08

Download references

Acknowledgements

This research was supported by the graduate scientific research and innovation project of Xinjiang Uygur Autonomous Region named “Research on in-depth learning of high school information technology courses for the cultivation of computing thinking” (No. XJ2022G190) and the independent innovation fund project for doctoral students of the College of Educational Science of Xinjiang Normal University named “Research on project-based teaching of high school information technology courses from the perspective of discipline core literacy” (No. XJNUJKYA2003).

Author information

Authors and affiliations.

College of Educational Science, Xinjiang Normal University, 830017, Urumqi, Xinjiang, China

Enwei Xu, Wei Wang & Qingxia Wang

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Enwei Xu or Wei Wang .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Informed consent

Additional information.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary tables, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Xu, E., Wang, W. & Wang, Q. The effectiveness of collaborative problem solving in promoting students’ critical thinking: A meta-analysis based on empirical literature. Humanit Soc Sci Commun 10 , 16 (2023). https://doi.org/10.1057/s41599-023-01508-1

Download citation

Received : 07 August 2022

Accepted : 04 January 2023

Published : 11 January 2023

DOI : https://doi.org/10.1057/s41599-023-01508-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Impacts of online collaborative learning on students’ intercultural communication apprehension and intercultural communicative competence.

• Hoa Thi Hoang Chau
• Hung Phu Bui
• Quynh Thi Huong Dinh

Education and Information Technologies (2024)

Exploring the effects of digital technology on deep learning: a meta-analysis

Sustainable electricity generation and farm-grid utilization from photovoltaic aquaculture: a bibliometric analysis.

• A. A. Amusa
• M. Alhassan

International Journal of Environmental Science and Technology (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Best Family Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Guided Meditations
• Verywell Mind Insights
• 2024 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

Overview of the Problem-Solving Mental Process

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Rachel Goldman, PhD FTOS, is a licensed psychologist, clinical assistant professor, speaker, wellness expert specializing in eating behaviors, stress management, and health behavior change.

• Identify the Problem
• Define the Problem
• Form a Strategy
• Organize Information
• Allocate Resources
• Monitor Progress
• Evaluate the Results

Frequently Asked Questions

Problem-solving is a mental process that involves discovering, analyzing, and solving problems. The ultimate goal of problem-solving is to overcome obstacles and find a solution that best resolves the issue.

The best strategy for solving a problem depends largely on the unique situation. In some cases, people are better off learning everything they can about the issue and then using factual knowledge to come up with a solution. In other instances, creativity and insight are the best options.

It is not necessary to follow problem-solving steps sequentially, It is common to skip steps or even go back through steps multiple times until the desired solution is reached.

In order to correctly solve a problem, it is often important to follow a series of steps. Researchers sometimes refer to this as the problem-solving cycle. While this cycle is portrayed sequentially, people rarely follow a rigid series of steps to find a solution.

The following steps include developing strategies and organizing knowledge.

1. Identifying the Problem

While it may seem like an obvious step, identifying the problem is not always as simple as it sounds. In some cases, people might mistakenly identify the wrong source of a problem, which will make attempts to solve it inefficient or even useless.

Some strategies that you might use to figure out the source of a problem include :

• Asking questions about the problem
• Breaking the problem down into smaller pieces
• Looking at the problem from different perspectives
• Conducting research to figure out what relationships exist between different variables

2. Defining the Problem

After the problem has been identified, it is important to fully define the problem so that it can be solved. You can define a problem by operationally defining each aspect of the problem and setting goals for what aspects of the problem you will address

At this point, you should focus on figuring out which aspects of the problems are facts and which are opinions. State the problem clearly and identify the scope of the solution.

3. Forming a Strategy

After the problem has been identified, it is time to start brainstorming potential solutions. This step usually involves generating as many ideas as possible without judging their quality. Once several possibilities have been generated, they can be evaluated and narrowed down.

The next step is to develop a strategy to solve the problem. The approach used will vary depending upon the situation and the individual's unique preferences. Common problem-solving strategies include heuristics and algorithms.

• Heuristics are mental shortcuts that are often based on solutions that have worked in the past. They can work well if the problem is similar to something you have encountered before and are often the best choice if you need a fast solution.
• Algorithms are step-by-step strategies that are guaranteed to produce a correct result. While this approach is great for accuracy, it can also consume time and resources.

Heuristics are often best used when time is of the essence, while algorithms are a better choice when a decision needs to be as accurate as possible.

4. Organizing Information

Before coming up with a solution, you need to first organize the available information. What do you know about the problem? What do you not know? The more information that is available the better prepared you will be to come up with an accurate solution.

When approaching a problem, it is important to make sure that you have all the data you need. Making a decision without adequate information can lead to biased or inaccurate results.

5. Allocating Resources

Of course, we don't always have unlimited money, time, and other resources to solve a problem. Before you begin to solve a problem, you need to determine how high priority it is.

If it is an important problem, it is probably worth allocating more resources to solving it. If, however, it is a fairly unimportant problem, then you do not want to spend too much of your available resources on coming up with a solution.

At this stage, it is important to consider all of the factors that might affect the problem at hand. This includes looking at the available resources, deadlines that need to be met, and any possible risks involved in each solution. After careful evaluation, a decision can be made about which solution to pursue.

6. Monitoring Progress

After selecting a problem-solving strategy, it is time to put the plan into action and see if it works. This step might involve trying out different solutions to see which one is the most effective.

It is also important to monitor the situation after implementing a solution to ensure that the problem has been solved and that no new problems have arisen as a result of the proposed solution.

Effective problem-solvers tend to monitor their progress as they work towards a solution. If they are not making good progress toward reaching their goal, they will reevaluate their approach or look for new strategies .

7. Evaluating the Results

After a solution has been reached, it is important to evaluate the results to determine if it is the best possible solution to the problem. This evaluation might be immediate, such as checking the results of a math problem to ensure the answer is correct, or it can be delayed, such as evaluating the success of a therapy program after several months of treatment.

Once a problem has been solved, it is important to take some time to reflect on the process that was used and evaluate the results. This will help you to improve your problem-solving skills and become more efficient at solving future problems.

A Word From Verywell​

It is important to remember that there are many different problem-solving processes with different steps, and this is just one example. Problem-solving in real-world situations requires a great deal of resourcefulness, flexibility, resilience, and continuous interaction with the environment.

Get Advice From The Verywell Mind Podcast

Hosted by therapist Amy Morin, LCSW, this episode of The Verywell Mind Podcast shares how you can stop dwelling in a negative mindset.

Follow Now : Apple Podcasts / Spotify / Google Podcasts

You can become a better problem solving by:

• Practicing brainstorming and coming up with multiple potential solutions to problems
• Being open-minded and considering all possible options before making a decision
• Breaking down problems into smaller, more manageable pieces
• Asking for help when needed
• Researching different problem-solving techniques and trying out new ones
• Learning from mistakes and using them as opportunities to grow

It's important to communicate openly and honestly with your partner about what's going on. Try to see things from their perspective as well as your own. Work together to find a resolution that works for both of you. Be willing to compromise and accept that there may not be a perfect solution.

Take breaks if things are getting too heated, and come back to the problem when you feel calm and collected. Don't try to fix every problem on your own—consider asking a therapist or counselor for help and insight.

If you've tried everything and there doesn't seem to be a way to fix the problem, you may have to learn to accept it. This can be difficult, but try to focus on the positive aspects of your life and remember that every situation is temporary. Don't dwell on what's going wrong—instead, think about what's going right. Find support by talking to friends or family. Seek professional help if you're having trouble coping.

Davidson JE, Sternberg RJ, editors.  The Psychology of Problem Solving .  Cambridge University Press; 2003. doi:10.1017/CBO9780511615771

Sarathy V. Real world problem-solving .  Front Hum Neurosci . 2018;12:261. Published 2018 Jun 26. doi:10.3389/fnhum.2018.00261

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

How to improve your problem solving skills and build effective problem solving strategies

Design your next session with SessionLab

Join the 150,000+ facilitators 
using SessionLab.

Recommended Articles

A step-by-step guide to planning a workshop, how to create an unforgettable training session in 8 simple steps, 47 useful online tools for workshop planning and meeting facilitation.

Effective problem solving is all about using the right process and following a plan tailored to the issue at hand. Recognizing your team or organization has an issue isn’t enough to come up with effective problem solving strategies. 

To truly understand a problem and develop appropriate solutions, you will want to follow a solid process, follow the necessary problem solving steps, and bring all of your problem solving skills to the table.  

We’ll first guide you through the seven step problem solving process you and your team can use to effectively solve complex business challenges. We’ll also look at what problem solving strategies you can employ with your team when looking for a way to approach the process. We’ll then discuss the problem solving skills you need to be more effective at solving problems, complete with an activity from the SessionLab library you can use to develop that skill in your team.

Let’s get to it! 

What is a problem solving process?

• What are the problem solving steps I need to follow?

Problem solving strategies

What skills do i need to be an effective problem solver, how can i improve my problem solving skills.

Solving problems is like baking a cake. You can go straight into the kitchen without a recipe or the right ingredients and do your best, but the end result is unlikely to be very tasty!

Using a process to bake a cake allows you to use the best ingredients without waste, collect the right tools, account for allergies, decide whether it is a birthday or wedding cake, and then bake efficiently and on time. The result is a better cake that is fit for purpose, tastes better and has created less mess in the kitchen. Also, it should have chocolate sprinkles. Having a step by step process to solve organizational problems allows you to go through each stage methodically and ensure you are trying to solve the right problems and select the most appropriate, effective solutions.

What are the problem solving steps I need to follow? 

All problem solving processes go through a number of steps in order to move from identifying a problem to resolving it.

Depending on your problem solving model and who you ask, there can be anything between four and nine problem solving steps you should follow in order to find the right solution. Whatever framework you and your group use, there are some key items that should be addressed in order to have an effective process.

We’ve looked at problem solving processes from sources such as the American Society for Quality and their four step approach , and Mediate ‘s six step process. By reflecting on those and our own problem solving processes, we’ve come up with a sequence of seven problem solving steps we feel best covers everything you need in order to effectively solve problems.

1. Problem identification 

The first stage of any problem solving process is to identify the problem or problems you might want to solve. Effective problem solving strategies always begin by allowing a group scope to articulate what they believe the problem to be and then coming to some consensus over which problem they approach first. Problem solving activities used at this stage often have a focus on creating frank, open discussion so that potential problems can be brought to the surface.

2. Problem analysis 

Though this step is not a million miles from problem identification, problem analysis deserves to be considered separately. It can often be an overlooked part of the process and is instrumental when it comes to developing effective solutions.

The process of problem analysis means ensuring that the problem you are seeking to solve is the right problem . As part of this stage, you may look deeper and try to find the root cause of a specific problem at a team or organizational level.

Remember that problem solving strategies should not only be focused on putting out fires in the short term but developing long term solutions that deal with the root cause of organizational challenges. 

Whatever your approach, analyzing a problem is crucial in being able to select an appropriate solution and the problem solving skills deployed in this stage are beneficial for the rest of the process and ensuring the solutions you create are fit for purpose.

3. Solution generation

Once your group has nailed down the particulars of the problem you wish to solve, you want to encourage a free flow of ideas connecting to solving that problem. This can take the form of problem solving games that encourage creative thinking or problem solving activities designed to produce working prototypes of possible solutions. 

The key to ensuring the success of this stage of the problem solving process is to encourage quick, creative thinking and create an open space where all ideas are considered. The best solutions can come from unlikely places and by using problem solving techniques that celebrate invention, you might come up with solution gold. 

4. Solution development

No solution is likely to be perfect right out of the gate. It’s important to discuss and develop the solutions your group has come up with over the course of following the previous problem solving steps in order to arrive at the best possible solution. Problem solving games used in this stage involve lots of critical thinking, measuring potential effort and impact, and looking at possible solutions analytically. 

During this stage, you will often ask your team to iterate and improve upon your frontrunning solutions and develop them further. Remember that problem solving strategies always benefit from a multitude of voices and opinions, and not to let ego get involved when it comes to choosing which solutions to develop and take further.

Finding the best solution is the goal of all problem solving workshops and here is the place to ensure that your solution is well thought out, sufficiently robust and fit for purpose. 

5. Decision making 

Nearly there! Once your group has reached consensus and selected a solution that applies to the problem at hand you have some decisions to make. You will want to work on allocating ownership of the project, figure out who will do what, how the success of the solution will be measured and decide the next course of action.

The decision making stage is a part of the problem solving process that can get missed or taken as for granted. Fail to properly allocate roles and plan out how a solution will actually be implemented and it less likely to be successful in solving the problem.

Have clear accountabilities, actions, timeframes, and follow-ups. Make these decisions and set clear next-steps in the problem solving workshop so that everyone is aligned and you can move forward effectively as a group. 

Ensuring that you plan for the roll-out of a solution is one of the most important problem solving steps. Without adequate planning or oversight, it can prove impossible to measure success or iterate further if the problem was not solved. 

6. Solution implementation 

This is what we were waiting for! All problem solving strategies have the end goal of implementing a solution and solving a problem in mind. 

Remember that in order for any solution to be successful, you need to help your group through all of the previous problem solving steps thoughtfully. Only then can you ensure that you are solving the right problem but also that you have developed the correct solution and can then successfully implement and measure the impact of that solution.

Project management and communication skills are key here – your solution may need to adjust when out in the wild or you might discover new challenges along the way.

7. Solution evaluation 

So you and your team developed a great solution to a problem and have a gut feeling its been solved. Work done, right? Wrong. All problem solving strategies benefit from evaluation, consideration, and feedback. You might find that the solution does not work for everyone, might create new problems, or is potentially so successful that you will want to roll it out to larger teams or as part of other initiatives. 

None of that is possible without taking the time to evaluate the success of the solution you developed in your problem solving model and adjust if necessary.

Remember that the problem solving process is often iterative and it can be common to not solve complex issues on the first try. Even when this is the case, you and your team will have generated learning that will be important for future problem solving workshops or in other parts of the organization. 

It’s worth underlining how important record keeping is throughout the problem solving process. If a solution didn’t work, you need to have the data and records to see why that was the case. If you go back to the drawing board, notes from the previous workshop can help save time. Data and insight is invaluable at every stage of the problem solving process and this one is no different.

Problem solving workshops made easy

Problem solving strategies are methods of approaching and facilitating the process of problem-solving with a set of techniques , actions, and processes. Different strategies are more effective if you are trying to solve broad problems such as achieving higher growth versus more focused problems like, how do we improve our customer onboarding process?

Broadly, the problem solving steps outlined above should be included in any problem solving strategy though choosing where to focus your time and what approaches should be taken is where they begin to differ. You might find that some strategies ask for the problem identification to be done prior to the session or that everything happens in the course of a one day workshop.

The key similarity is that all good problem solving strategies are structured and designed. Four hours of open discussion is never going to be as productive as a four-hour workshop designed to lead a group through a problem solving process.

Good problem solving strategies are tailored to the team, organization and problem you will be attempting to solve. Here are some example problem solving strategies you can learn from or use to get started.

Use a workshop to lead a team through a group process

Often, the first step to solving problems or organizational challenges is bringing a group together effectively. Most teams have the tools, knowledge, and expertise necessary to solve their challenges – they just need some guidance in how to use leverage those skills and a structure and format that allows people to focus their energies.

Facilitated workshops are one of the most effective ways of solving problems of any scale. By designing and planning your workshop carefully, you can tailor the approach and scope to best fit the needs of your team and organization. 

Problem solving workshop

• Creating a bespoke, tailored process
• Tackling problems of any size
• Building in-house workshop ability and encouraging their use

Workshops are an effective strategy for solving problems. By using tried and test facilitation techniques and methods, you can design and deliver a workshop that is perfectly suited to the unique variables of your organization. You may only have the capacity for a half-day workshop and so need a problem solving process to match. 

By using our session planner tool and importing methods from our library of 700+ facilitation techniques, you can create the right problem solving workshop for your team. It might be that you want to encourage creative thinking or look at things from a new angle to unblock your groups approach to problem solving. By tailoring your workshop design to the purpose, you can help ensure great results.

One of the main benefits of a workshop is the structured approach to problem solving. Not only does this mean that the workshop itself will be successful, but many of the methods and techniques will help your team improve their working processes outside of the workshop. 

We believe that workshops are one of the best tools you can use to improve the way your team works together. Start with a problem solving workshop and then see what team building, culture or design workshops can do for your organization!

Run a design sprint

Great for: 

• aligning large, multi-discipline teams
• quickly designing and testing solutions
• tackling large, complex organizational challenges and breaking them down into smaller tasks

By using design thinking principles and methods, a design sprint is a great way of identifying, prioritizing and prototyping solutions to long term challenges that can help solve major organizational problems with quick action and measurable results.

Some familiarity with design thinking is useful, though not integral, and this strategy can really help a team align if there is some discussion around which problems should be approached first. 

The stage-based structure of the design sprint is also very useful for teams new to design thinking.  The inspiration phase, where you look to competitors that have solved your problem, and the rapid prototyping and testing phases are great for introducing new concepts that will benefit a team in all their future work. 

It can be common for teams to look inward for solutions and so looking to the market for solutions you can iterate on can be very productive. Instilling an agile prototyping and testing mindset can also be great when helping teams move forwards – generating and testing solutions quickly can help save time in the long run and is also pretty exciting!

Break problems down into smaller issues

Organizational challenges and problems are often complicated and large scale in nature. Sometimes, trying to resolve such an issue in one swoop is simply unachievable or overwhelming. Try breaking down such problems into smaller issues that you can work on step by step. You may not be able to solve the problem of churning customers off the bat, but you can work with your team to identify smaller effort but high impact elements and work on those first.

This problem solving strategy can help a team generate momentum, prioritize and get some easy wins. It’s also a great strategy to employ with teams who are just beginning to learn how to approach the problem solving process. If you want some insight into a way to employ this strategy, we recommend looking at our design sprint template below!

Use guiding frameworks or try new methodologies

Some problems are best solved by introducing a major shift in perspective or by using new methodologies that encourage your team to think differently.

Props and tools such as Methodkit , which uses a card-based toolkit for facilitation, or Lego Serious Play can be great ways to engage your team and find an inclusive, democratic problem solving strategy. Remember that play and creativity are great tools for achieving change and whatever the challenge, engaging your participants can be very effective where other strategies may have failed.

LEGO Serious Play

• Improving core problem solving skills
• Thinking outside of the box
• Encouraging creative solutions

LEGO Serious Play is a problem solving methodology designed to get participants thinking differently by using 3D models and kinesthetic learning styles. By physically building LEGO models based on questions and exercises, participants are encouraged to think outside of the box and create their own responses. 

Collaborate LEGO Serious Play exercises are also used to encourage communication and build problem solving skills in a group. By using this problem solving process, you can often help different kinds of learners and personality types contribute and unblock organizational problems with creative thinking. 

Problem solving strategies like LEGO Serious Play are super effective at helping a team solve more skills-based problems such as communication between teams or a lack of creative thinking. Some problems are not suited to LEGO Serious Play and require a different problem solving strategy.

Card Decks and Method Kits

• New facilitators or non-facilitators 
• Approaching difficult subjects with a simple, creative framework
• Engaging those with varied learning styles

Card decks and method kids are great tools for those new to facilitation or for whom facilitation is not the primary role. Card decks such as the emotional culture deck can be used for complete workshops and in many cases, can be used right out of the box. Methodkit has a variety of kits designed for scenarios ranging from personal development through to personas and global challenges so you can find the right deck for your particular needs.

Having an easy to use framework that encourages creativity or a new approach can take some of the friction or planning difficulties out of the workshop process and energize a team in any setting. Simplicity is the key with these methods. By ensuring everyone on your team can get involved and engage with the process as quickly as possible can really contribute to the success of your problem solving strategy.

Source external advice

Looking to peers, experts and external facilitators can be a great way of approaching the problem solving process. Your team may not have the necessary expertise, insights of experience to tackle some issues, or you might simply benefit from a fresh perspective. Some problems may require bringing together an entire team, and coaching managers or team members individually might be the right approach. Remember that not all problems are best resolved in the same manner.

If you’re a solo entrepreneur, peer groups, coaches and mentors can also be invaluable at not only solving specific business problems, but in providing a support network for resolving future challenges. One great approach is to join a Mastermind Group and link up with like-minded individuals and all grow together. Remember that however you approach the sourcing of external advice, do so thoughtfully, respectfully and honestly. Reciprocate where you can and prepare to be surprised by just how kind and helpful your peers can be!

Mastermind Group

• Solo entrepreneurs or small teams with low capacity
• Peer learning and gaining outside expertise
• Getting multiple external points of view quickly

Problem solving in large organizations with lots of skilled team members is one thing, but how about if you work for yourself or in a very small team without the capacity to get the most from a design sprint or LEGO Serious Play session? 

A mastermind group – sometimes known as a peer advisory board – is where a group of people come together to support one another in their own goals, challenges, and businesses. Each participant comes to the group with their own purpose and the other members of the group will help them create solutions, brainstorm ideas, and support one another. 

Mastermind groups are very effective in creating an energized, supportive atmosphere that can deliver meaningful results. Learning from peers from outside of your organization or industry can really help unlock new ways of thinking and drive growth. Access to the experience and skills of your peers can be invaluable in helping fill the gaps in your own ability, particularly in young companies.

A mastermind group is a great solution for solo entrepreneurs, small teams, or for organizations that feel that external expertise or fresh perspectives will be beneficial for them. It is worth noting that Mastermind groups are often only as good as the participants and what they can bring to the group. Participants need to be committed, engaged and understand how to work in this context. 

Coaching and mentoring

• Focused learning and development
• Filling skills gaps
• Working on a range of challenges over time

Receiving advice from a business coach or building a mentor/mentee relationship can be an effective way of resolving certain challenges. The one-to-one format of most coaching and mentor relationships can really help solve the challenges those individuals are having and benefit the organization as a result.

A great mentor can be invaluable when it comes to spotting potential problems before they arise and coming to understand a mentee very well has a host of other business benefits. You might run an internal mentorship program to help develop your team’s problem solving skills and strategies or as part of a large learning and development program. External coaches can also be an important part of your problem solving strategy, filling skills gaps for your management team or helping with specific business issues. 

Now we’ve explored the problem solving process and the steps you will want to go through in order to have an effective session, let’s look at the skills you and your team need to be more effective problem solvers.

Problem solving skills are highly sought after, whatever industry or team you work in. Organizations are keen to employ people who are able to approach problems thoughtfully and find strong, realistic solutions. Whether you are a facilitator , a team leader or a developer, being an effective problem solver is a skill you’ll want to develop.

Problem solving skills form a whole suite of techniques and approaches that an individual uses to not only identify problems but to discuss them productively before then developing appropriate solutions.

Here are some of the most important problem solving skills everyone from executives to junior staff members should learn. We’ve also included an activity or exercise from the SessionLab library that can help you and your team develop that skill. 

If you’re running a workshop or training session to try and improve problem solving skills in your team, try using these methods to supercharge your process!

Active listening

Active listening is one of the most important skills anyone who works with people can possess. In short, active listening is a technique used to not only better understand what is being said by an individual, but also to be more aware of the underlying message the speaker is trying to convey. When it comes to problem solving, active listening is integral for understanding the position of every participant and to clarify the challenges, ideas and solutions they bring to the table.

Some active listening skills include:

• Paying complete attention to the speaker.
• Removing distractions.
• Avoid interruption.
• Taking the time to fully understand before preparing a rebuttal.
• Responding respectfully and appropriately.
• Demonstrate attentiveness and positivity with an open posture, making eye contact with the speaker, smiling and nodding if appropriate. Show that you are listening and encourage them to continue.
• Be aware of and respectful of feelings. Judge the situation and respond appropriately. You can disagree without being disrespectful.   
• Observe body language. 
• Paraphrase what was said in your own words, either mentally or verbally.
• Remain neutral. 
• Reflect and take a moment before responding.
• Ask deeper questions based on what is said and clarify points where necessary.   

Active Listening   #hyperisland   #skills   #active listening   #remote-friendly   This activity supports participants to reflect on a question and generate their own solutions using simple principles of active listening and peer coaching. It’s an excellent introduction to active listening but can also be used with groups that are already familiar with it. Participants work in groups of three and take turns being: “the subject”, the listener, and the observer.

Analytical skills

All problem solving models require strong analytical skills, particularly during the beginning of the process and when it comes to analyzing how solutions have performed.

Analytical skills are primarily focused on performing an effective analysis by collecting, studying and parsing data related to a problem or opportunity. 

It often involves spotting patterns, being able to see things from different perspectives and using observable facts and data to make suggestions or produce insight. 

Analytical skills are also important at every stage of the problem solving process and by having these skills, you can ensure that any ideas or solutions you create or backed up analytically and have been sufficiently thought out.

Nine Whys   #innovation   #issue analysis   #liberating structures   With breathtaking simplicity, you can rapidly clarify for individuals and a group what is essentially important in their work. You can quickly reveal when a compelling purpose is missing in a gathering and avoid moving forward without clarity. When a group discovers an unambiguous shared purpose, more freedom and more responsibility are unleashed. You have laid the foundation for spreading and scaling innovations with fidelity.

Collaboration

Trying to solve problems on your own is difficult. Being able to collaborate effectively, with a free exchange of ideas, to delegate and be a productive member of a team is hugely important to all problem solving strategies.

Remember that whatever your role, collaboration is integral, and in a problem solving process, you are all working together to find the best solution for everyone. 

Marshmallow challenge with debriefing   #teamwork   #team   #leadership   #collaboration   In eighteen minutes, teams must build the tallest free-standing structure out of 20 sticks of spaghetti, one yard of tape, one yard of string, and one marshmallow. The marshmallow needs to be on top. The Marshmallow Challenge was developed by Tom Wujec, who has done the activity with hundreds of groups around the world. Visit the Marshmallow Challenge website for more information. This version has an extra debriefing question added with sample questions focusing on roles within the team.

Communication  

Being an effective communicator means being empathetic, clear and succinct, asking the right questions, and demonstrating active listening skills throughout any discussion or meeting. 

In a problem solving setting, you need to communicate well in order to progress through each stage of the process effectively. As a team leader, it may also fall to you to facilitate communication between parties who may not see eye to eye. Effective communication also means helping others to express themselves and be heard in a group.

Bus Trip   #feedback   #communication   #appreciation   #closing   #thiagi   #team   This is one of my favourite feedback games. I use Bus Trip at the end of a training session or a meeting, and I use it all the time. The game creates a massive amount of energy with lots of smiles, laughs, and sometimes even a teardrop or two.

Creative problem solving skills can be some of the best tools in your arsenal. Thinking creatively, being able to generate lots of ideas and come up with out of the box solutions is useful at every step of the process. 

The kinds of problems you will likely discuss in a problem solving workshop are often difficult to solve, and by approaching things in a fresh, creative manner, you can often create more innovative solutions.

Having practical creative skills is also a boon when it comes to problem solving. If you can help create quality design sketches and prototypes in record time, it can help bring a team to alignment more quickly or provide a base for further iteration.

The paper clip method   #sharing   #creativity   #warm up   #idea generation   #brainstorming   The power of brainstorming. A training for project leaders, creativity training, and to catalyse getting new solutions.

Critical thinking

Critical thinking is one of the fundamental problem solving skills you’ll want to develop when working on developing solutions. Critical thinking is the ability to analyze, rationalize and evaluate while being aware of personal bias, outlying factors and remaining open-minded.

Defining and analyzing problems without deploying critical thinking skills can mean you and your team go down the wrong path. Developing solutions to complex issues requires critical thinking too – ensuring your team considers all possibilities and rationally evaluating them. 

Agreement-Certainty Matrix   #issue analysis   #liberating structures   #problem solving   You can help individuals or groups avoid the frequent mistake of trying to solve a problem with methods that are not adapted to the nature of their challenge. The combination of two questions makes it possible to easily sort challenges into four categories: simple, complicated, complex , and chaotic .  A problem is simple when it can be solved reliably with practices that are easy to duplicate.  It is complicated when experts are required to devise a sophisticated solution that will yield the desired results predictably.  A problem is complex when there are several valid ways to proceed but outcomes are not predictable in detail.  Chaotic is when the context is too turbulent to identify a path forward.  A loose analogy may be used to describe these differences: simple is like following a recipe, complicated like sending a rocket to the moon, complex like raising a child, and chaotic is like the game “Pin the Tail on the Donkey.”  The Liberating Structures Matching Matrix in Chapter 5 can be used as the first step to clarify the nature of a challenge and avoid the mismatches between problems and solutions that are frequently at the root of chronic, recurring problems.

Data analysis 

Though it shares lots of space with general analytical skills, data analysis skills are something you want to cultivate in their own right in order to be an effective problem solver.

Being good at data analysis doesn’t just mean being able to find insights from data, but also selecting the appropriate data for a given issue, interpreting it effectively and knowing how to model and present that data. Depending on the problem at hand, it might also include a working knowledge of specific data analysis tools and procedures. 

Having a solid grasp of data analysis techniques is useful if you’re leading a problem solving workshop but if you’re not an expert, don’t worry. Bring people into the group who has this skill set and help your team be more effective as a result.

Decision making

All problems need a solution and all solutions require that someone make the decision to implement them. Without strong decision making skills, teams can become bogged down in discussion and less effective as a result. 

Making decisions is a key part of the problem solving process. It’s important to remember that decision making is not restricted to the leadership team. Every staff member makes decisions every day and developing these skills ensures that your team is able to solve problems at any scale. Remember that making decisions does not mean leaping to the first solution but weighing up the options and coming to an informed, well thought out solution to any given problem that works for the whole team.

Lightning Decision Jam (LDJ)   #action   #decision making   #problem solving   #issue analysis   #innovation   #design   #remote-friendly   The problem with anything that requires creative thinking is that it’s easy to get lost—lose focus and fall into the trap of having useless, open-ended, unstructured discussions. Here’s the most effective solution I’ve found: Replace all open, unstructured discussion with a clear process. What to use this exercise for: Anything which requires a group of people to make decisions, solve problems or discuss challenges. It’s always good to frame an LDJ session with a broad topic, here are some examples: The conversion flow of our checkout Our internal design process How we organise events Keeping up with our competition Improving sales flow

Dependability

Most complex organizational problems require multiple people to be involved in delivering the solution. Ensuring that the team and organization can depend on you to take the necessary actions and communicate where necessary is key to ensuring problems are solved effectively.

Being dependable also means working to deadlines and to brief. It is often a matter of creating trust in a team so that everyone can depend on one another to complete the agreed actions in the agreed time frame so that the team can move forward together. Being undependable can create problems of friction and can limit the effectiveness of your solutions so be sure to bear this in mind throughout a project. 

Team Purpose & Culture   #team   #hyperisland   #culture   #remote-friendly   This is an essential process designed to help teams define their purpose (why they exist) and their culture (how they work together to achieve that purpose). Defining these two things will help any team to be more focused and aligned. With support of tangible examples from other companies, the team members work as individuals and a group to codify the way they work together. The goal is a visual manifestation of both the purpose and culture that can be put up in the team’s work space.

Emotional intelligence

Emotional intelligence is an important skill for any successful team member, whether communicating internally or with clients or users. In the problem solving process, emotional intelligence means being attuned to how people are feeling and thinking, communicating effectively and being self-aware of what you bring to a room. 

There are often differences of opinion when working through problem solving processes, and it can be easy to let things become impassioned or combative. Developing your emotional intelligence means being empathetic to your colleagues and managing your own emotions throughout the problem and solution process. Be kind, be thoughtful and put your points across care and attention. 

Being emotionally intelligent is a skill for life and by deploying it at work, you can not only work efficiently but empathetically. Check out the emotional culture workshop template for more!

Facilitation

As we’ve clarified in our facilitation skills post, facilitation is the art of leading people through processes towards agreed-upon objectives in a manner that encourages participation, ownership, and creativity by all those involved. While facilitation is a set of interrelated skills in itself, the broad definition of facilitation can be invaluable when it comes to problem solving. Leading a team through a problem solving process is made more effective if you improve and utilize facilitation skills – whether you’re a manager, team leader or external stakeholder.

The Six Thinking Hats   #creative thinking   #meeting facilitation   #problem solving   #issue resolution   #idea generation   #conflict resolution   The Six Thinking Hats are used by individuals and groups to separate out conflicting styles of thinking. They enable and encourage a group of people to think constructively together in exploring and implementing change, rather than using argument to fight over who is right and who is wrong.

Flexibility 

Being flexible is a vital skill when it comes to problem solving. This does not mean immediately bowing to pressure or changing your opinion quickly: instead, being flexible is all about seeing things from new perspectives, receiving new information and factoring it into your thought process.

Flexibility is also important when it comes to rolling out solutions. It might be that other organizational projects have greater priority or require the same resources as your chosen solution. Being flexible means understanding needs and challenges across the team and being open to shifting or arranging your own schedule as necessary. Again, this does not mean immediately making way for other projects. It’s about articulating your own needs, understanding the needs of others and being able to come to a meaningful compromise.

The Creativity Dice   #creativity   #problem solving   #thiagi   #issue analysis   Too much linear thinking is hazardous to creative problem solving. To be creative, you should approach the problem (or the opportunity) from different points of view. You should leave a thought hanging in mid-air and move to another. This skipping around prevents premature closure and lets your brain incubate one line of thought while you consciously pursue another.

Working in any group can lead to unconscious elements of groupthink or situations in which you may not wish to be entirely honest. Disagreeing with the opinions of the executive team or wishing to save the feelings of a coworker can be tricky to navigate, but being honest is absolutely vital when to comes to developing effective solutions and ensuring your voice is heard. 

Remember that being honest does not mean being brutally candid. You can deliver your honest feedback and opinions thoughtfully and without creating friction by using other skills such as emotional intelligence. 

Explore your Values   #hyperisland   #skills   #values   #remote-friendly   Your Values is an exercise for participants to explore what their most important values are. It’s done in an intuitive and rapid way to encourage participants to follow their intuitive feeling rather than over-thinking and finding the “correct” values. It is a good exercise to use to initiate reflection and dialogue around personal values.

Initiative 

The problem solving process is multi-faceted and requires different approaches at certain points of the process. Taking initiative to bring problems to the attention of the team, collect data or lead the solution creating process is always valuable. You might even roadtest your own small scale solutions or brainstorm before a session. Taking initiative is particularly effective if you have good deal of knowledge in that area or have ownership of a particular project and want to get things kickstarted.

That said, be sure to remember to honor the process and work in service of the team. If you are asked to own one part of the problem solving process and you don’t complete that task because your initiative leads you to work on something else, that’s not an effective method of solving business challenges.

15% Solutions   #action   #liberating structures   #remote-friendly   You can reveal the actions, however small, that everyone can do immediately. At a minimum, these will create momentum, and that may make a BIG difference.  15% Solutions show that there is no reason to wait around, feel powerless, or fearful. They help people pick it up a level. They get individuals and the group to focus on what is within their discretion instead of what they cannot change.  With a very simple question, you can flip the conversation to what can be done and find solutions to big problems that are often distributed widely in places not known in advance. Shifting a few grains of sand may trigger a landslide and change the whole landscape.

Impartiality

A particularly useful problem solving skill for product owners or managers is the ability to remain impartial throughout much of the process. In practice, this means treating all points of view and ideas brought forward in a meeting equally and ensuring that your own areas of interest or ownership are not favored over others. 

There may be a stage in the process where a decision maker has to weigh the cost and ROI of possible solutions against the company roadmap though even then, ensuring that the decision made is based on merit and not personal opinion. 

Empathy map   #frame insights   #create   #design   #issue analysis   An empathy map is a tool to help a design team to empathize with the people they are designing for. You can make an empathy map for a group of people or for a persona. To be used after doing personas when more insights are needed.

Being a good leader means getting a team aligned, energized and focused around a common goal. In the problem solving process, strong leadership helps ensure that the process is efficient, that any conflicts are resolved and that a team is managed in the direction of success.

It’s common for managers or executives to assume this role in a problem solving workshop, though it’s important that the leader maintains impartiality and does not bulldoze the group in a particular direction. Remember that good leadership means working in service of the purpose and team and ensuring the workshop is a safe space for employees of any level to contribute. Take a look at our leadership games and activities post for more exercises and methods to help improve leadership in your organization.

Leadership Pizza   #leadership   #team   #remote-friendly   This leadership development activity offers a self-assessment framework for people to first identify what skills, attributes and attitudes they find important for effective leadership, and then assess their own development and initiate goal setting.

In the context of problem solving, mediation is important in keeping a team engaged, happy and free of conflict. When leading or facilitating a problem solving workshop, you are likely to run into differences of opinion. Depending on the nature of the problem, certain issues may be brought up that are emotive in nature. 

Being an effective mediator means helping those people on either side of such a divide are heard, listen to one another and encouraged to find common ground and a resolution. Mediating skills are useful for leaders and managers in many situations and the problem solving process is no different.

Conflict Responses   #hyperisland   #team   #issue resolution   A workshop for a team to reflect on past conflicts, and use them to generate guidelines for effective conflict handling. The workshop uses the Thomas-Killman model of conflict responses to frame a reflective discussion. Use it to open up a discussion around conflict with a team.

Planning 

Solving organizational problems is much more effective when following a process or problem solving model. Planning skills are vital in order to structure, deliver and follow-through on a problem solving workshop and ensure your solutions are intelligently deployed.

Planning skills include the ability to organize tasks and a team, plan and design the process and take into account any potential challenges. Taking the time to plan carefully can save time and frustration later in the process and is valuable for ensuring a team is positioned for success.

3 Action Steps   #hyperisland   #action   #remote-friendly   This is a small-scale strategic planning session that helps groups and individuals to take action toward a desired change. It is often used at the end of a workshop or programme. The group discusses and agrees on a vision, then creates some action steps that will lead them towards that vision. The scope of the challenge is also defined, through discussion of the helpful and harmful factors influencing the group.

Prioritization

As organisations grow, the scale and variation of problems they face multiplies. Your team or is likely to face numerous challenges in different areas and so having the skills to analyze and prioritize becomes very important, particularly for those in leadership roles.

A thorough problem solving process is likely to deliver multiple solutions and you may have several different problems you wish to solve simultaneously. Prioritization is the ability to measure the importance, value, and effectiveness of those possible solutions and choose which to enact and in what order. The process of prioritization is integral in ensuring the biggest challenges are addressed with the most impactful solutions.

Impact and Effort Matrix   #gamestorming   #decision making   #action   #remote-friendly   In this decision-making exercise, possible actions are mapped based on two factors: effort required to implement and potential impact. Categorizing ideas along these lines is a useful technique in decision making, as it obliges contributors to balance and evaluate suggested actions before committing to them.

Project management

Some problem solving skills are utilized in a workshop or ideation phases, while others come in useful when it comes to decision making. Overseeing an entire problem solving process and ensuring its success requires strong project management skills. 

While project management incorporates many of the other skills listed here, it is important to note the distinction of considering all of the factors of a project and managing them successfully. Being able to negotiate with stakeholders, manage tasks, time and people, consider costs and ROI, and tie everything together is massively helpful when going through the problem solving process. 

Record keeping

Working out meaningful solutions to organizational challenges is only one part of the process.  Thoughtfully documenting and keeping records of each problem solving step for future consultation is important in ensuring efficiency and meaningful change. 

For example, some problems may be lower priority than others but can be revisited in the future. If the team has ideated on solutions and found some are not up to the task, record those so you can rule them out and avoiding repeating work. Keeping records of the process also helps you improve and refine your problem solving model next time around!

Personal Kanban   #gamestorming   #action   #agile   #project planning   Personal Kanban is a tool for organizing your work to be more efficient and productive. It is based on agile methods and principles.

Research skills

Conducting research to support both the identification of problems and the development of appropriate solutions is important for an effective process. Knowing where to go to collect research, how to conduct research efficiently, and identifying pieces of research are relevant are all things a good researcher can do well. 

In larger groups, not everyone has to demonstrate this ability in order for a problem solving workshop to be effective. That said, having people with research skills involved in the process, particularly if they have existing area knowledge, can help ensure the solutions that are developed with data that supports their intention. Remember that being able to deliver the results of research efficiently and in a way the team can easily understand is also important. The best data in the world is only as effective as how it is delivered and interpreted.

Customer experience map   #ideation   #concepts   #research   #design   #issue analysis   #remote-friendly   Customer experience mapping is a method of documenting and visualizing the experience a customer has as they use the product or service. It also maps out their responses to their experiences. To be used when there is a solution (even in a conceptual stage) that can be analyzed.

Risk management

Managing risk is an often overlooked part of the problem solving process. Solutions are often developed with the intention of reducing exposure to risk or solving issues that create risk but sometimes, great solutions are more experimental in nature and as such, deploying them needs to be carefully considered. 

Managing risk means acknowledging that there may be risks associated with more out of the box solutions or trying new things, but that this must be measured against the possible benefits and other organizational factors. 

Be informed, get the right data and stakeholders in the room and you can appropriately factor risk into your decision making process. 

Decisions, Decisions…   #communication   #decision making   #thiagi   #action   #issue analysis   When it comes to decision-making, why are some of us more prone to take risks while others are risk-averse? One explanation might be the way the decision and options were presented.  This exercise, based on Kahneman and Tversky’s classic study , illustrates how the framing effect influences our judgement and our ability to make decisions . The participants are divided into two groups. Both groups are presented with the same problem and two alternative programs for solving them. The two programs both have the same consequences but are presented differently. The debriefing discussion examines how the framing of the program impacted the participant’s decision.

Team-building 

No single person is as good at problem solving as a team. Building an effective team and helping them come together around a common purpose is one of the most important problem solving skills, doubly so for leaders. By bringing a team together and helping them work efficiently, you pave the way for team ownership of a problem and the development of effective solutions. 

In a problem solving workshop, it can be tempting to jump right into the deep end, though taking the time to break the ice, energize the team and align them with a game or exercise will pay off over the course of the day.

Remember that you will likely go through the problem solving process multiple times over an organization’s lifespan and building a strong team culture will make future problem solving more effective. It’s also great to work with people you know, trust and have fun with. Working on team building in and out of the problem solving process is a hallmark of successful teams that can work together to solve business problems.

9 Dimensions Team Building Activity   #ice breaker   #teambuilding   #team   #remote-friendly   9 Dimensions is a powerful activity designed to build relationships and trust among team members. There are 2 variations of this icebreaker. The first version is for teams who want to get to know each other better. The second version is for teams who want to explore how they are working together as a team.

Time management 

The problem solving process is designed to lead a team from identifying a problem through to delivering a solution and evaluating its effectiveness. Without effective time management skills or timeboxing of tasks, it can be easy for a team to get bogged down or be inefficient.

By using a problem solving model and carefully designing your workshop, you can allocate time efficiently and trust that the process will deliver the results you need in a good timeframe.

Time management also comes into play when it comes to rolling out solutions, particularly those that are experimental in nature. Having a clear timeframe for implementing and evaluating solutions is vital for ensuring their success and being able to pivot if necessary.

Improving your skills at problem solving is often a career-long pursuit though there are methods you can use to make the learning process more efficient and to supercharge your problem solving skillset.

Remember that the skills you need to be a great problem solver have a large overlap with those skills you need to be effective in any role. Investing time and effort to develop your active listening or critical thinking skills is valuable in any context. Here are 7 ways to improve your problem solving skills.

Share best practices

Remember that your team is an excellent source of skills, wisdom, and techniques and that you should all take advantage of one another where possible. Best practices that one team has for solving problems, conducting research or making decisions should be shared across the organization. If you have in-house staff that have done active listening training or are data analysis pros, have them lead a training session. 

Your team is one of your best resources. Create space and internal processes for the sharing of skills so that you can all grow together. 

Ask for help and attend training

Once you’ve figured out you have a skills gap, the next step is to take action to fill that skills gap. That might be by asking your superior for training or coaching, or liaising with team members with that skill set. You might even attend specialized training for certain skills – active listening or critical thinking, for example, are business-critical skills that are regularly offered as part of a training scheme.

Whatever method you choose, remember that taking action of some description is necessary for growth. Whether that means practicing, getting help, attending training or doing some background reading, taking active steps to improve your skills is the way to go.

Learn a process 

Problem solving can be complicated, particularly when attempting to solve large problems for the first time. Using a problem solving process helps give structure to your problem solving efforts and focus on creating outcomes, rather than worrying about the format. 

Tools such as the seven-step problem solving process above are effective because not only do they feature steps that will help a team solve problems, they also develop skills along the way. Each step asks for people to engage with the process using different skills and in doing so, helps the team learn and grow together. Group processes of varying complexity and purpose can also be found in the SessionLab library of facilitation techniques . Using a tried and tested process and really help ease the learning curve for both those leading such a process, as well as those undergoing the purpose.

Effective teams make decisions about where they should and shouldn’t expend additional effort. By using a problem solving process, you can focus on the things that matter, rather than stumbling towards a solution haphazardly. 

Create a feedback loop

Some skills gaps are more obvious than others. It’s possible that your perception of your active listening skills differs from those of your colleagues. 

It’s valuable to create a system where team members can provide feedback in an ordered and friendly manner so they can all learn from one another. Only by identifying areas of improvement can you then work to improve them. 

Remember that feedback systems require oversight and consideration so that they don’t turn into a place to complain about colleagues. Design the system intelligently so that you encourage the creation of learning opportunities, rather than encouraging people to list their pet peeves.

While practice might not make perfect, it does make the problem solving process easier. If you are having trouble with critical thinking, don’t shy away from doing it. Get involved where you can and stretch those muscles as regularly as possible. 

Problem solving skills come more naturally to some than to others and that’s okay. Take opportunities to get involved and see where you can practice your skills in situations outside of a workshop context. Try collaborating in other circumstances at work or conduct data analysis on your own projects. You can often develop those skills you need for problem solving simply by doing them. Get involved!

Use expert exercises and methods

Learn from the best. Our library of 700+ facilitation techniques is full of activities and methods that help develop the skills you need to be an effective problem solver. Check out our templates to see how to approach problem solving and other organizational challenges in a structured and intelligent manner.

There is no single approach to improving problem solving skills, but by using the techniques employed by others you can learn from their example and develop processes that have seen proven results. 

Try new ways of thinking and change your mindset

Using tried and tested exercises that you know well can help deliver results, but you do run the risk of missing out on the learning opportunities offered by new approaches. As with the problem solving process, changing your mindset can remove blockages and be used to develop your problem solving skills.

Most teams have members with mixed skill sets and specialties. Mix people from different teams and share skills and different points of view. Teach your customer support team how to use design thinking methods or help your developers with conflict resolution techniques. Try switching perspectives with facilitation techniques like Flip It! or by using new problem solving methodologies or models. Give design thinking, liberating structures or lego serious play a try if you want to try a new approach. You will find that framing problems in new ways and using existing skills in new contexts can be hugely useful for personal development and improving your skillset. It’s also a lot of fun to try new things. Give it a go!

Encountering business challenges and needing to find appropriate solutions is not unique to your organization. Lots of very smart people have developed methods, theories and approaches to help develop problem solving skills and create effective solutions. Learn from them!

Books like The Art of Thinking Clearly , Think Smarter, or Thinking Fast, Thinking Slow are great places to start, though it’s also worth looking at blogs related to organizations facing similar problems to yours, or browsing for success stories. Seeing how Dropbox massively increased growth and working backward can help you see the skills or approach you might be lacking to solve that same problem. Learning from others by reading their stories or approaches can be time-consuming but ultimately rewarding.

A tired, distracted mind is not in the best position to learn new skills. It can be tempted to burn the candle at both ends and develop problem solving skills outside of work. Absolutely use your time effectively and take opportunities for self-improvement, though remember that rest is hugely important and that without letting your brain rest, you cannot be at your most effective. 

Creating distance between yourself and the problem you might be facing can also be useful. By letting an idea sit, you can find that a better one presents itself or you can develop it further. Take regular breaks when working and create a space for downtime. Remember that working smarter is preferable to working harder and that self-care is important for any effective learning or improvement process.

Want to design better group processes?

Over to you

Now we’ve explored some of the key problem solving skills and the problem solving steps necessary for an effective process, you’re ready to begin developing more effective solutions and leading problem solving workshops.

Need more inspiration? Check out our post on problem solving activities you can use when guiding a group towards a great solution in your next workshop or meeting. Have questions? Did you have a great problem solving technique you use with your team? Get in touch in the comments below. We’d love to chat!

Leave a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Going from a mere idea to a workshop that delivers results for your clients can feel like a daunting task. In this piece, we will shine a light on all the work behind the scenes and help you learn how to plan a workshop from start to finish. On a good day, facilitation can feel like effortless magic, but that is mostly the result of backstage work, foresight, and a lot of careful planning. Read on to learn a step-by-step approach to breaking the process of planning a workshop into small, manageable chunks.  The flow starts with the first meeting with a client to define the purposes of a workshop.…

How does learning work? A clever 9-year-old once told me: “I know I am learning something new when I am surprised.” The science of adult learning tells us that, in order to learn new skills (which, unsurprisingly, is harder for adults to do than kids) grown-ups need to first get into a specific headspace.  In a business, this approach is often employed in a training session where employees learn new skills or work on professional development. But how do you ensure your training is effective? In this guide, we'll explore how to create an effective training session plan and run engaging training sessions. As team leader, project manager, or consultant,…

Effective online tools are a necessity for smooth and engaging virtual workshops and meetings. But how do you choose the right ones? Do you sometimes feel that the good old pen and paper or MS Office toolkit and email leaves you struggling to stay on top of managing and delivering your workshop? Fortunately, there are plenty of online tools to make your life easier when you need to facilitate a meeting and lead workshops. In this post, we’ll share our favorite online tools you can use to make your job as a facilitator easier. In fact, there are plenty of free online workshop tools and meeting facilitation software you can…

Design your next workshop with SessionLab

Join the 150,000 facilitators using SessionLab

Sign up for free

• Prodigy Math
• Prodigy English
• Is a Premium Membership Worth It?
• Promote a Growth Mindset
• Help Your Child Who's Struggling with Math
• Parent's Guide to Prodigy
• Assessments
• Math Curriculum Coverage
• English Curriculum Coverage
• Game Portal

5 Advantages and Disadvantages of Problem-Based Learning [+ Activity Design Steps]

Written by Marcus Guido

Easily differentiate learning and engage your students with Prodigy Math.

• Teaching Strategies

Advantages of Problem-Based Learning

Disadvantages of problem-based learning, steps to designing problem-based learning activities.

Used since the 1960s, many teachers express concerns about the effectiveness of problem-based learning (PBL) in certain classroom settings.

Whether you introduce the student-centred pedagogy as a one-time activity or mainstay exercise, grouping students together to solve open-ended problems can present pros and cons.

Below are five advantages and disadvantages of problem-based learning to help you determine if it can work in your classroom.

If you decide to introduce an activity, there are also design creation steps and a downloadable guide to keep at your desk for easy reference.

1. Development of Long-Term Knowledge Retention

Students who participate in problem-based learning activities can improve their abilities to retain and recall information, according to a literature review of studies about the pedagogy .

The literature review states “elaboration of knowledge at the time of learning” -- by sharing facts and ideas through discussion and answering questions -- “enhances subsequent retrieval.” This form of elaborating reinforces understanding of subject matter , making it easier to remember.

Small-group discussion can be especially beneficial -- ideally, each student will get chances to participate.

But regardless of group size, problem-based learning promotes long-term knowledge retention by encouraging students to discuss -- and answer questions about -- new concepts as they’re learning them.

2. Use of Diverse Instruction Types

You can use problem-based learning activities to the meet the diverse learning needs and styles of your students, effectively engaging a diverse classroom in the process. In general, grouping students together for problem-based learning will allow them to:

• Address real-life issues that require real-life solutions, appealing to students who struggle to grasp abstract concepts
• Participate in small-group and large-group learning, helping students who don’t excel during solo work grasp new material
• Talk about their ideas and challenge each other in a constructive manner, giving participatory learners an avenue to excel
• Tackle a problem using a range of content you provide -- such as videos, audio recordings, news articles and other applicable material -- allowing the lesson to appeal to distinct learning styles

Since running a problem-based learning scenario will give you a way to use these differentiated instruction approaches , it can be especially worthwhile if your students don’t have similar learning preferences.

3. Continuous Engagement

Providing a problem-based learning challenge can engage students by acting as a break from normal lessons and common exercises.

It’s not hard to see the potential for engagement, as kids collaborate to solve real-world problems that directly affect or heavily interest them.

Although conducted with post-secondary students, a study published by the Association for the Study of Medical Education reported increased student attendance to -- and better attitudes towards -- courses that feature problem-based learning.

These activities may lose some inherent engagement if you repeat them too often, but can certainly inject excitement into class.

4. Development of Transferable Skills

Problem-based learning can help students develop skills they can transfer to real-world scenarios, according to a 2015 book that outlines theories and characteristics of the pedagogy .

The tangible contexts and consequences presented in a problem-based learning activity “allow learning to become more profound and durable.” As you present lessons through these real-life scenarios, students should be able to apply learnings if they eventually face similar issues.

For example, if they work together to address a dispute within the school, they may develop lifelong skills related to negotiation and communicating their thoughts with others.

As long as the problem’s context applies to out-of-class scenarios, students should be able to build skills they can use again.

5. Improvement of Teamwork and Interpersonal Skills

Successful completion of a problem-based learning challenge hinges on interaction and communication, meaning students should also build transferable skills based on teamwork and collaboration . Instead of memorizing facts, they get chances to present their ideas to a group, defending and revising them when needed.

What’s more, this should help them understand a group dynamic. Depending on a given student, this can involve developing listening skills and a sense of responsibility when completing one’s tasks. Such skills and knowledge should serve your students well when they enter higher education levels and, eventually, the working world.

1. Potentially Poorer Performance on Tests

Devoting too much time to problem-based learning can cause issues when students take standardized tests, as they may not have the breadth of knowledge needed to achieve high scores. Whereas problem-based learners develop skills related to collaboration and justifying their reasoning, many tests reward fact-based learning with multiple choice and short answer questions. Despite offering many advantages, you could spot this problem develop if you run problem-based learning activities too regularly.

2. Student Unpreparedness

Problem-based learning exercises can engage many of your kids, but others may feel disengaged as a result of not being ready to handle this type of exercise for a number of reasons. On a class-by-class and activity-by-activity basis, participation may be hindered due to:

• Immaturity  -- Some students may not display enough maturity to effectively work in a group, not fulfilling expectations and distracting other students.
• Unfamiliarity  -- Some kids may struggle to grasp the concept of an open problem, since they can’t rely on you for answers.
• Lack of Prerequisite Knowledge  -- Although the activity should address a relevant and tangible problem, students may require new or abstract information to create an effective solution.

You can partially mitigate these issues by actively monitoring the classroom and distributing helpful resources, such as guiding questions and articles to read. This should keep students focused and help them overcome knowledge gaps. But if you foresee facing these challenges too frequently, you may decide to avoid or seldom introduce problem-based learning exercises.

3. Teacher Unpreparedness

If supervising a problem-based learning activity is a new experience, you may have to prepare to adjust some teaching habits . For example, overtly correcting students who make flawed assumptions or statements can prevent them from thinking through difficult concepts and questions. Similarly, you shouldn’t teach to promote the fast recall of facts. Instead, you should concentrate on:

• Giving hints to help fix improper reasoning
• Questioning student logic and ideas in a constructive manner
• Distributing content for research and to reinforce new concepts
• Asking targeted questions to a group or the class, focusing their attention on a specific aspect of the problem

Depending on your teaching style, it may take time to prepare yourself to successfully run a problem-based learning lesson.

4. Time-Consuming Assessment

If you choose to give marks, assessing a student’s performance throughout a problem-based learning exercise demands constant monitoring and note-taking. You must take factors into account such as:

• Completed tasks
• The quality of those tasks
• The group’s overall work and solution
• Communication among team members
• Anything you outlined on the activity’s rubric

Monitoring these criteria is required for each student, making it time-consuming to give and justify a mark for everyone.

5. Varying Degrees of Relevancy and Applicability

It can be difficult to identify a tangible problem that students can solve with content they’re studying and skills they’re mastering. This introduces two clear issues. First, if it is easy for students to divert from the challenge’s objectives, they may miss pertinent information. Second, you could veer off the problem’s focus and purpose as students run into unanticipated obstacles. Overcoming obstacles has benefits, but may compromise the planning you did. It can also make it hard to get back on track once the activity is complete. Because of the difficulty associated with keeping activities relevant and applicable, you may see problem-based learning as too taxing.

If the advantages outweigh the disadvantages -- or you just want to give problem-based learning a shot -- follow these steps:

1. Identify an Applicable Real-Life Problem

Find a tangible problem that’s relevant to your students, allowing them to easily contextualize it and hopefully apply it to future challenges. To identify an appropriate real-world problem, look at issues related to your:

• Students’ shared interests

You must also ensure that students understand the problem and the information around it. So, not all problems are appropriate for all grade levels.

2. Determine the Overarching Purpose of the Activity

Depending on the problem you choose, determine what you want to accomplish by running the challenge. For example, you may intend to help your students improve skills related to:

• Collaboration
• Problem-solving
• Curriculum-aligned topics
• Processing diverse content

A more precise example, you may prioritize collaboration skills by assigning specific tasks to pairs of students within each team. In doing so, students will continuously develop communication and collaboration abilities by working as a couple and part of a small group. By defining a clear purpose, you’ll also have an easier time following the next step.

3. Create and Distribute Helpful Material

Handouts and other content not only act as a set of resources, but help students stay focused on the activity and its purpose. For example, if you want them to improve a certain math skill , you should make material that highlights the mathematical aspects of the problem. You may decide to provide items such as:

• Data that helps quantify and add context to the problem
• Videos, presentations and other audio-visual material
• A list of preliminary questions to investigate

Providing a range of resources can be especially important for elementary students and struggling students in higher grades, who may not have self-direction skills to work without them.

4. Set Goals and Expectations for Your Students

Along with the aforementioned materials, give students a guide or rubric that details goals and expectations. It will allow you to further highlight the purpose of the problem-based learning exercise, as you can explain what you’re looking for in terms of collaboration, the final product and anything else. It should also help students stay on track by acting as a reference throughout the activity.

5. Participate

Although explicitly correcting students may be discouraged, you can still help them and ask questions to dig into their thought processes. When you see an opportunity, consider if it’s worthwhile to:

• Fill gaps in knowledge
• Provide hints, not answers
• Question a student’s conclusion or logic regarding a certain point, helping them think through tough spots

By participating in these ways, you can provide insight when students need it most, encouraging them to effectively analyze the problem.

6. Have Students Present Ideas and Findings

If you divided them into small groups, requiring students to present their thoughts and results in front the class adds a large-group learning component to the lesson. Encourage other students to ask questions, allowing the presenting group to elaborate and provide evidence for their thoughts. This wraps up the activity and gives your class a final chance to find solutions to the problem.

Wrapping Up

The effectiveness of problem-based learning may differ between classrooms and individual students, depending on how significant specific advantages and disadvantages are to you. Evaluative research consistently shows value in giving students a question and letting them take control of their learning. But the extent of this value can depend on the difficulties you face.It may be wise to try a problem-based learning activity, and go forward based on results.

Create or log into your teacher account on Prodigy -- an adaptive math game that adjusts content to accommodate player trouble spots and learning speeds. Aligned to US and Canadian curricula, it’s used by more than 350,000 teachers and 10 million students. It may be wise to try a problem-based learning activity, and go forward based on results.

• Business Essentials
• Leadership & Management
• Credential of Leadership, Impact, and Management in Business (CLIMB)
• Entrepreneurship & Innovation
• Digital Transformation
• Finance & Accounting
• Business in Society
• For Organizations
• Support Portal
• Media Coverage
• Founding Donors
• Leadership Team

• Harvard Business School →
• HBS Online →
• Business Insights →

Business Insights

Harvard Business School Online's Business Insights Blog provides the career insights you need to achieve your goals and gain confidence in your business skills.

• Career Development
• Communication
• Decision-Making
• Earning Your MBA
• Negotiation
• News & Events
• Productivity
• Staff Spotlight
• Student Profiles
• Work-Life Balance
• AI Essentials for Business
• Alternative Investments
• Business Analytics
• Business Strategy
• Business and Climate Change
• Design Thinking and Innovation
• Digital Marketing Strategy
• Disruptive Strategy
• Economics for Managers
• Entrepreneurship Essentials
• Financial Accounting
• Global Business
• Launching Tech Ventures
• Leadership Principles
• Leadership, Ethics, and Corporate Accountability
• Leading with Finance
• Management Essentials
• Negotiation Mastery
• Organizational Leadership
• Power and Influence for Positive Impact
• Strategy Execution
• Sustainable Business Strategy
• Sustainable Investing
• Winning with Digital Platforms

What Is Creative Problem-Solving & Why Is It Important?

• 01 Feb 2022

One of the biggest hindrances to innovation is complacency—it can be more comfortable to do what you know than venture into the unknown. Business leaders can overcome this barrier by mobilizing creative team members and providing space to innovate.

There are several tools you can use to encourage creativity in the workplace. Creative problem-solving is one of them, which facilitates the development of innovative solutions to difficult problems.

Here’s an overview of creative problem-solving and why it’s important in business.

Access your free e-book today.

What Is Creative Problem-Solving?

Research is necessary when solving a problem. But there are situations where a problem’s specific cause is difficult to pinpoint. This can occur when there’s not enough time to narrow down the problem’s source or there are differing opinions about its root cause.

In such cases, you can use creative problem-solving , which allows you to explore potential solutions regardless of whether a problem has been defined.

Creative problem-solving is less structured than other innovation processes and encourages exploring open-ended solutions. It also focuses on developing new perspectives and fostering creativity in the workplace . Its benefits include:

• Finding creative solutions to complex problems : User research can insufficiently illustrate a situation’s complexity. While other innovation processes rely on this information, creative problem-solving can yield solutions without it.
• Adapting to change : Business is constantly changing, and business leaders need to adapt. Creative problem-solving helps overcome unforeseen challenges and find solutions to unconventional problems.
• Fueling innovation and growth : In addition to solutions, creative problem-solving can spark innovative ideas that drive company growth. These ideas can lead to new product lines, services, or a modified operations structure that improves efficiency.

Creative problem-solving is traditionally based on the following key principles :

1. Balance Divergent and Convergent Thinking

Creative problem-solving uses two primary tools to find solutions: divergence and convergence. Divergence generates ideas in response to a problem, while convergence narrows them down to a shortlist. It balances these two practices and turns ideas into concrete solutions.

2. Reframe Problems as Questions

By framing problems as questions, you shift from focusing on obstacles to solutions. This provides the freedom to brainstorm potential ideas.

3. Defer Judgment of Ideas

When brainstorming, it can be natural to reject or accept ideas right away. Yet, immediate judgments interfere with the idea generation process. Even ideas that seem implausible can turn into outstanding innovations upon further exploration and development.

4. Focus on "Yes, And" Instead of "No, But"

Using negative words like "no" discourages creative thinking. Instead, use positive language to build and maintain an environment that fosters the development of creative and innovative ideas.

Creative Problem-Solving and Design Thinking

Whereas creative problem-solving facilitates developing innovative ideas through a less structured workflow, design thinking takes a far more organized approach.

Design thinking is a human-centered, solutions-based process that fosters the ideation and development of solutions. In the online course Design Thinking and Innovation , Harvard Business School Dean Srikant Datar leverages a four-phase framework to explain design thinking.

The four stages are:

• Clarify: The clarification stage allows you to empathize with the user and identify problems. Observations and insights are informed by thorough research. Findings are then reframed as problem statements or questions.
• Ideate: Ideation is the process of coming up with innovative ideas. The divergence of ideas involved with creative problem-solving is a major focus.
• Develop: In the development stage, ideas evolve into experiments and tests. Ideas converge and are explored through prototyping and open critique.
• Implement: Implementation involves continuing to test and experiment to refine the solution and encourage its adoption.

Creative problem-solving primarily operates in the ideate phase of design thinking but can be applied to others. This is because design thinking is an iterative process that moves between the stages as ideas are generated and pursued. This is normal and encouraged, as innovation requires exploring multiple ideas.

Creative Problem-Solving Tools

While there are many useful tools in the creative problem-solving process, here are three you should know:

Creating a Problem Story

One way to innovate is by creating a story about a problem to understand how it affects users and what solutions best fit their needs. Here are the steps you need to take to use this tool properly.

1. Identify a UDP

Create a problem story to identify the undesired phenomena (UDP). For example, consider a company that produces printers that overheat. In this case, the UDP is "our printers overheat."

2. Move Forward in Time

To move forward in time, ask: “Why is this a problem?” For example, minor damage could be one result of the machines overheating. In more extreme cases, printers may catch fire. Don't be afraid to create multiple problem stories if you think of more than one UDP.

3. Move Backward in Time

To move backward in time, ask: “What caused this UDP?” If you can't identify the root problem, think about what typically causes the UDP to occur. For the overheating printers, overuse could be a cause.

Following the three-step framework above helps illustrate a clear problem story:

• The printer is overused.
• The printer overheats.
• The printer breaks down.

You can extend the problem story in either direction if you think of additional cause-and-effect relationships.

4. Break the Chains

By this point, you’ll have multiple UDP storylines. Take two that are similar and focus on breaking the chains connecting them. This can be accomplished through inversion or neutralization.

• Inversion: Inversion changes the relationship between two UDPs so the cause is the same but the effect is the opposite. For example, if the UDP is "the more X happens, the more likely Y is to happen," inversion changes the equation to "the more X happens, the less likely Y is to happen." Using the printer example, inversion would consider: "What if the more a printer is used, the less likely it’s going to overheat?" Innovation requires an open mind. Just because a solution initially seems unlikely doesn't mean it can't be pursued further or spark additional ideas.
• Neutralization: Neutralization completely eliminates the cause-and-effect relationship between X and Y. This changes the above equation to "the more or less X happens has no effect on Y." In the case of the printers, neutralization would rephrase the relationship to "the more or less a printer is used has no effect on whether it overheats."

Even if creating a problem story doesn't provide a solution, it can offer useful context to users’ problems and additional ideas to be explored. Given that divergence is one of the fundamental practices of creative problem-solving, it’s a good idea to incorporate it into each tool you use.

Brainstorming

Brainstorming is a tool that can be highly effective when guided by the iterative qualities of the design thinking process. It involves openly discussing and debating ideas and topics in a group setting. This facilitates idea generation and exploration as different team members consider the same concept from multiple perspectives.

Hosting brainstorming sessions can result in problems, such as groupthink or social loafing. To combat this, leverage a three-step brainstorming method involving divergence and convergence :

• Have each group member come up with as many ideas as possible and write them down to ensure the brainstorming session is productive.
• Continue the divergence of ideas by collectively sharing and exploring each idea as a group. The goal is to create a setting where new ideas are inspired by open discussion.
• Begin the convergence of ideas by narrowing them down to a few explorable options. There’s no "right number of ideas." Don't be afraid to consider exploring all of them, as long as you have the resources to do so.

Alternate Worlds

The alternate worlds tool is an empathetic approach to creative problem-solving. It encourages you to consider how someone in another world would approach your situation.

For example, if you’re concerned that the printers you produce overheat and catch fire, consider how a different industry would approach the problem. How would an automotive expert solve it? How would a firefighter?

Be creative as you consider and research alternate worlds. The purpose is not to nail down a solution right away but to continue the ideation process through diverging and exploring ideas.

Continue Developing Your Skills

Whether you’re an entrepreneur, marketer, or business leader, learning the ropes of design thinking can be an effective way to build your skills and foster creativity and innovation in any setting.

If you're ready to develop your design thinking and creative problem-solving skills, explore Design Thinking and Innovation , one of our online entrepreneurship and innovation courses. If you aren't sure which course is the right fit, download our free course flowchart to determine which best aligns with your goals.

About the Author

Math Fluency in Problem Solving, Grades 3, 4, & 5

Math Fluency in Problem Solving

Accelerate student learning and growth with Math Fluency in Problems Solving!

Take a look at our digital workbook!

Principal Testimonials!

Call 888-902-3005 or [email protected] to order PAPERBACK books. Digital books can be ordered above in our store NOW! PURCHASE ORDERS ONLY for paperback books!

We provide thousands of word problems in our 3rd, 4th, and 5th grade paperback & digital math books. our books will have your students over-prepared.

Word Problems, Basic Facts, & Math Computations all in one book! 888-902-3005 [email protected]

Johnson Learning Systems

Math fluency in problem solving 3rd, 4th & 5th grade, quick links.

• Teacher Login
• Student Login

Machine Unlearning in 2024

41 minute read

Written by Ken Liu ∙ May 2024

1. A bit of history & motivations for unlearning

2.1. exact unlearning.

• 2.2. "Unlearning" via differential privacy
• 2.3. Empirical unlearning with known example space
• 2.4. Empirical unlearning with unknown example space

2.5. Just ask for unlearning?

3. evaluating unlearning, 4.1. the spectrum of unlearning hardness, 4.2. copyright protection, 4.3. retrieval-based ai systems, 4.4. ai safety.

As our ML models today become larger and their (pre-)training sets grow to inscrutable sizes, people are increasingly interested in the concept of machine unlearning to edit away undesired things like private data, stale knowledge, copyrighted materials, toxic/unsafe content, dangerous capabilities, and misinformation, without retraining models from scratch.

Machine unlearning can be broadly described as removing the influences of training data from a trained model. At its core, unlearning on a target model seeks to produce an unlearned model that is equivalent to—or at least “behaves like”—a retrained model that is trained on the same data of target model, minus the information to be unlearned.

There’s a lot hidden in the above description. How do we describe the information to be unlearned? Do we always have ground-truth retrained models? If not, how do we actually evaluate the unlearning? Can we even verify and audit the unlearning? Is pretending to unlearn, as humans often do, sufficient? Is unlearning even the right solution? If so, for what problems?

The precise definitions of unlearning, the techniques, the guarantees, and the metrics/evaluations would depend on:

• The ML task (e.g., binary classification or language modeling);
• The data to unlearn (e.g., a set of images, news articles, or the knowledge of making napalm );
• The unlearning algorithm (e.g., heuristic fine-tuning vs deleting model components);
• The goal of unlearning (e.g., for user privacy or harmfulness removal).

In this educational post, I hope to give a gentle, general ML audience introduction to machine unlearning and touch on things like copyright protection , New York Times v. OpenAI , right-to-be-forgotten , NeurIPS machine unlearning challenge , retrieval-based AI systems , AI safety , along with some of my thoughts on the field. While unlearning is broad topic applicable to most ML models, we will focus a lot on foundation models .

People have thought about the unlearning problem for a while now . The initial research explorations were primarily driven by Article 17 of GDPR (European Union’s privacy regulation), often referred to as “ right-to-be-forgotten ” ( RTBF ) since 2014. RTBF basically says a user has the right to request deletion of their data from a service provider (e.g. deleting your Gmail account).

RTBF was well-intentioned. It was also very actionable when said service providers store user data in a structured way, like how Google removed a bunch of links from its index in repsonse to RTBF requests.

However, RTBF wasn’t really proposed with machine learning in mind. In 2014, policymakers wouldn’t have predicted that deep learning will be a giant hodgepodge of data & compute, and that separating and interpreting this hodgepodge turned out to be hard. The hardness of erasing data from ML models has subsequently motivated research on what is later referred to as “ data deletion ” and “ machine unlearning ”.

A decade later in 2024, user privacy is no longer the only motivation for unlearning. We’ve gone from training small convolutional nets on face images to training giant language models on pay-walled, copyrighted , toxic , dangerous, and otherwise harmful content, all of which we may want to “erase” from the ML models—sometimes with access to only a handful of examples. The nature of the models has changed too. Instead of using many small specialized models each good at one task, people started using a single giant model that knows just about any task.

Currently, I think the motivations for unlearning fall into two categories:

Access revocation (think unlearning private and copyrighted data). In an ideal world, data should be thought of as “borrowed” (possibly unpermittedly) and thus can be “returned”, and unlearning should enable such revocation.

Unlearning is challenging from this perspective. One key difficulty is that our limited understanding of deep learning itself makes data trained into a model akin to “consumables” (which can’t just be “returned” after consumption). Data may also be non-fungible (e.g. your chat history) and may even be thought of as labor with its own financial and control interests. Another challenge is that access revocation may require a proof of unlearning; as we will explore in the coming sections, this isn’t always possible.

These difficulties suggest that it’s perhaps also worth revising laws like RTBF and thinking about alternatives such as data markets , where data owners are properly compensated so they won’t want to request unlearning in the first place. To illustrate, suppose Bob ate Alice’s cheesecake (data), Alice would much rather Bob pay her or return something equivalent (compensation) than Bob puking to his pre-eating state (unlearning).

In practice, one way to implement access revocation is via some form of periodic re-training of the base model. Many model providers already do this to keep their models competitive and up-to-date. For example, OpenAI can collect a bunch of unlearning requests, and batch-satisfy them during the re-training every year (or, guided by RTBF’s “ undue delay ” period by which the request must be satisfied). More broadly, this suggests socio-technical solutions for unlearning: policymakers can mandate such periodic re-training and set economically viable deadlines to offload the costs to the model owners.

Model correction & editing (think toxicity, bias, stale/dangerous knowledge removal). That is, the model was trained on something undesirable and we’d like to fix it. This is closely related to the model editing literature. The concept of “ corrective machine unlearning ”, where unlearning serves to correct the impact of bad data, was recently proposed to capture this motivation. From this perspective, unlearning may also be viewed as a post-training risk mitigation mechanism for AI safety concerns (discussed further in Section 4).

Unlike access revocation, we could be more lenient towards with model correction since the edit is more of a desire than a necessity mandated by law, much like model accuracy on image classification or toxicity of generated text. (Of course, these can cause real harm too.) Here, we won’t necessarily need formal guarantees for the unlearning to be practically useful; we have plenty of examples where people would happily deploy models that are deemed “sufficiently safe”. The recent WMDP benchmark , which quizzes a model on hazardous knowledge, is a good example of empirically evaluating unlearning efficacy.

2. Forms of unlearning

Unlearning is trivially satisfied if we can just retrain the model without the undesired data. However, we want something better because (1) retraining can be expensive and (2) it can be a lot of work just to find out what to remove from training data—think finding all Harry Potter references in a trillion tokens. Unlearning techniques essentially seek to mitigate or avoid this retraining cost while producing identical or similar results.

The unlearning literature can roughly be categorized into the following:

• Exact unlearning
• “Unlearning” via differential privacy
• Empirical unlearning, where data to be unlearned are precisely known (training examples)
• Empirical unlearning, where data to be unlearned are underspecified (think “knowledge”)
• Just ask for unlearning?

Forms 2-4 are sometimes known as “ approximate unlearning ” in that the unlearned model approximates the behavior of the retrained model. Form 5 is quite new and interesting, and more specific to instruction-following models.

In the following, we will go through what each of these types roughly looks like, along with what I think are the promises, caveats, and questions to ask looking forward.

Exact unlearning roughly asks that the unlearned model and the retrained model to be distributionally identical ; that is, they can be exactly the same under fixed randomness.

Techniques for exact unlearning are characterized by the early work of Cao & Yang and SISA . In SISA, a very simple scheme, the training set is split into $N$ non-overlapping subsets, and a separate model is trained for each subset. Unlearning involves retraining the model corresponding to and without the data points to be unlearned. This reduces cost from vanilla retraining by $1/N$ (cheaper if we keep model checkpoints). Inference then involves model ensembling. 1

More generally, the essence of exact unlearning of this form is that we want modular components in the learning algorithm to correspond to different (potentially disjoint) sets of the training examples.

There are several benefits of exact unlearning:

• The algorithm is the proof . If we implement something like SISA, we know by design that the unlearned data never contributed to other components. As it turns out, formally proving the model has unlearned something is quite challenging otherwise.
• It turns the unlearning problem into an accuracy/efficiency problem. This makes exact unlearning more approachable due to the messiness of unlearning evaluation and lack of benchmarks.
• Interpretability by design . By providing a structure to learning, we also have better understanding of how certain data points contribute to performance.

The main drawback seems obvious: modern scaling law of large models argues against excessive data & model sharding as done in SISA. Or does it? I think it would be very interesting to revisit sharding in the context of large models, in light of the recent model merging literature that suggests the feasibility of weight-space merging between large models. As we’ll learn in the coming sections, the messiness of approximate unlearning and its evaluation, especially in the context of large models, makes exact unlearning very appealing.

2.2. “Unlearning” via differential privacy

This line of work roughly says: if the model behaves more or less the same with or without any particular data point, then there’s nothing we need to unlearn from that data point. More broadly, we are asking for distributional closeness between the unlearned and the retrained models.

For readers unfamilar with differential privacy (DP) in machine learning, DP defines a quantifiable indistinguishability guarantee between two models $M$, $M’$ trained on datasets $X$, $X’$ that differ in any single training example. The canonical procedure, DP-SGD , works by clipping the L2-norm of the per-example gradients and injecting some per-coordinate Gaussian noise to the gradients. The idea is that the noise would mask or obscure the contribution of any single gradient (example), such that the final model isn’t sensitive any exmaple. It is usually denoted by ($\varepsilon, \delta$)-DP; the stronger the noise, the smaller the scalars ($\varepsilon, \delta$), the more private.

The intuition is that if an adversary cannot (reliably) tell apart the models, then it is as if this data point has never been learned—thus no need to unlearn. DP can be used to achieve this form of unlearning, but due to the one-sidedness of unlearning (where we only care about data removal, not addition), DP is a strictly stronger definition . This notion of unlearning is sometimes known as “ ($\alpha, \beta$)-unlearning ” where ($\alpha, \beta$) serve similar roles as ($\varepsilon, \delta$) to measure distributional closeness.

Example techniques along this direction include: (1) storing checkpoints of (DP) convex models and unlearning is retraining from those checkpoints; and (2) on top of the previous technique, add SISA for adaptive unlearning requests (i.e. those that come in after observing the published model).

DP-based unlearning is good in that it gives some form of a statistical guarantee. However, there are some important considerations that limit its applicability to large models :

• Many such unlearning results apply only to convex models or losses .
• What levels of unlearning (values of $(\varepsilon, \delta)$-DP or $(\alpha, \beta)$-unlearning) are sufficient? Who decides?
• For large models, current ML systems don’t fit well with the per-example workloads of DP-like procedures. The memory overhead will also be prohibitive.
• Moreoever, like DP, the guarantees can fall off quickly with more unlearning requests (at best the rate of $O(\sqrt k)$ with $k$ requests following DP composition theorems).
• DP-like definitions implicitly assume we care about all data points equally . But some examples are more likely to receive unlearning request, and some examples would not have contributed to the learning at all.
• DP-like procedures may also just hurt model accuracy a lot, sometimes in an unfair way.

For large models in particular, it’s also worth distinguishing the cases of unlearning pre-training data vs unlearning fine-tuning data . The latter is a lot more tractable; for example, we could indeed fine-tune large models with differential privacy but not so much with pre-training.

2.2.1. Forging and its implications on DP-like unlearning definitions

An unlearning procedure may sometimes require an external audit , meaning that we’d like to prove that the unlearning procedure has actually happened.

The main idea of “ forging ” is that there exists two distinct datasets that, when trained on, would produce the same gradients and (thus) the same models . This is true intuitively:

• Think linear regression of points on a perfect line; removing any 1 point doesn’t change the fitted line;
• Think mini-batch GD, where replacing one example gradient with the sum of several “fake” gradients would give the same batch gradient.

Forging implies that DP-based approximate unlearning may not be auditable —that is, the unlearning service provider cannot formally prove that the forget set is really forgotten. In fact, if we only look at the model weights, even exact unlearning may not be auditable.

While one can brush this off as a theoretical result, it does mean that policymakers should think carefully about how a future version of “right-to-be-forgotten” (if any) should look like and whether similar policies are legally and technically enforceable.

Indeed, what qualifies as an “audit” could very well be definition and application dependent. If the auditor only cares that the unlearned model performs poorly on a specified set of inputs (say on a set of face images), then even empirical unlearning is “auditable” (see next section).

2.3. Empirical unlearning with known example space (“example unlearning”)

This line of work is essentially “training to unlearn” or “unlearning via fine-tuning”: just take a few more heuristically chosen gradient steps to shape the original model’s behavior into what we think the retrained model would do (while also optionally resetting some parameters in the model). It may also be referred to as “example unlearning”, since the training, retain, and forget sets are often clearly defined.

The NeurIPS 2023 Machine Unlearning Challenge collected many methods along this direction. The challenge roughly runs as follows:

• You are given a face image dataset with designated retain/forget example splits for the training set, a target model trained on everything, and a secret model trained only on the retain set.
• You are asked to design an unlearning algorithm that produces unlearned model(s) from the target model that “match” the secretly kept model.
• The “match” or evaluation metric uses a DP-like output-space similarity over 512 seeds: for each forget example, compute an “empirical $\varepsilon$” over 512 unlearned models based on true/false positive rates of an adversary (also provided by the organizer), and aggregate across examples.
• All models are a small ConvNet.

To give an intuition about how well empirical unlearning is doing without fully explaining the metric: the ground-truth retrained model gets about ~0.19, the winning submission gets to ~0.12, and the baseline (simple gradient ascent on forget set) is ~0.06. 2

So what do the winning ideas look like? Something along the lines of the following:

• Gradient ascent on the forget set;
• Gradient descent on the retain set (and hope that catastrophic forgetting takes care of unlearning);
• Gradient descent on the forget set, but with uniformly random labels (to “confuse” the model);
• Minimize KL divergence on outputs between unlearned model and original model on the retain set (to regularize unlearned model performance on unrelated data);
• Re-initialize weights that had similar gradients on the retain set and forget sets, and finetune these weights on the retain set;
• Prune 99% of weights by L1-norm and fine-tune on the retain set;
• Reset first/last $k$ layers and fine-tune on the retain set; and
• Heuristic/arbitrary combinations of the above.

Indeed, despite the heuristic nature of these approaches, these are what most empirical unlearning algorithms , especially those on large (language) models , are doing these days.

People explore empirical approaches because theoretical tools are usually impractical; for example, enforcing DP simply hurts accuracy and efficiency too much, even for the GPU rich. On the flip side, empirical methods are often fast and easy to implement, and their effects are often qualitatively visible.

Another key motivation for empirical unlearning is that counterfactuals are unclear, especially on LLMs. In deep learning, we often don’t know how the retrained model would behave on unseen data. What should the LLM think who Biden is, if not a politician? Should image classifiers give uniformly random predictions for unlearned images? Do they generalize? Or are they confidently wrong? Any of these is possible and it can be up to the practitioner to decide. It also means that behaviors that are equally plausible can lead to wildly different measurements (e.g., KL divergence between output distributions of unlearned & retrained model), complicating theoretical guarantees.

2.4. Empirical unlearning with unknown example space (“concept/knowledge unlearning”)

What if the train, retain, or forget sets are poorly specified or just not specified at all? Foundation models that train on internet-scale data may get requests to unlearn a “ concept ”, a “ fact ”, or a piece of “ knowledge ”, all of which we cannot easily associate a set of examples. The terms “ model editing ”, “ concept editing ”, “ model surgery ”, and “ knowledge unlearning ” are closely related to this notion of unlearning. 3

The underspecification of the unlearning requests means that we now have to deal with the notions of “ unlearning scope ” (or “ editing scope ”) and “ entailment ”. That is, unlearning requests may provide canonical examples to indicate what to unlearn, but the same information can manifest in the (pre-)training set in many different forms with many different downstream implications such that simply achieving unlearning on these examples—even exactly —would not suffice.

For example:

• The association “Biden is the US president” is dispersed throughout various forms of text from news articles, books, casual text messages, or this very blog post. Can we ever unlearn all occurrences? Moreover, does unlearning Joe Biden also entail unlearning the color of Biden’s cat ?
• Artists may request to unlearn art style by providing art samples, but they won’t be able to collect everything they have on the internet and their adaptations .
• New York Times may request to unlearn news articles, but they cannot enumerate quotes and secondary transformations of these articles.

Such vagueness also suggests that unlearning pre-training data from large models are perhaps necessarily empirical: it is unlikely to derive formal guarantees if we can’t clearly specify what to (and what not to) unlearn in the trillions of tokens and establish clear information boundaries between different entities. An interesting implication of achieving unlearning empirically is that the unlearning itself can be unlearned .

What does existing work do, then, with underspecified unlearning requests? Most techniques are more or less the same as before , except now we also need to find the examples to fine-tune on. For example, attempting to unlearn Harry Potter involves asking GPT-4 to come up with plausible alternative text completions (e.g. that Mr. Potter studies baking instead of magic); and attempting to unlearn harmful behavior involves collecting examples of hatespeech.

Another set of techniques involves training the desired behavior (or its opposite) into task / control vectors and harnessing the capability of large models to undergo weight-space merging or activation steering . The fundamental approach of the above is more or less the same, nevertheless—obtaining these edit vectors involves (heuristically) designing what gradients to take and what data on which to take them. One could also frame the unlearning problem as an alignment problem and applies the forget examples with a DPO-like objective .

It turns out that powerful, instruction-following LLMs like GPT-4 are smart enough to pretend to unlearn . This means crafting prompts to induce a (sufficiently) safe behavior for the target unlearning application.

This is an interesting approach because no gradients are involved whatsoever (big plus from a systems perspective), and intuitively the end results could very well be as good as existing empirical unlearning techniques. Among different ways we could prompt, past work explored the following two directions.

Literally asking to pretend unlearning. We can ask in the system prompt to, say, pretend to not know who Harry Potter is. By design, this works best for common entities, facts, knowledge, or behaviors (e.g. the ability to utter like Trump) that are well-captured in the pre-training set, since the LLM needs to know it well to pretend not knowing it well . On the other hand, suppose now we’d like to unlearn the address of an obscure person; the pre-training set is so large that we suspect it’s part of training data. We now face a variant of the Streisand effect : is it even worth asking the model to pretend unlearning by accurately describing it in-context, and subsequently risk leaking it in subsequent model responses?

Few-shot prompting or “ in-context unlearning ”. Suppose we now have a clearly defined set of forget examples with corresponding labels. We can flip their labels and put them in the prompt, along with more retain examples with correct labels, with the intuition that the model would treat these falsely labelled forget examples as truths and act accordingly—much like one could jailbreak a model this way. 4 Indeed, this works best when the forget examples and the counterfactual labels are clearly defined and (somewhat) finite. It may work for factual associations (e.g. Paris is the captial of France) by enumerating a lot of examples, but unlikely to work for unlearning toxic behaviors (where space of possible outputs is much larger).

In a sense, these approaches are complementary as they work for different kinds of unlearning requests.

More broadly, one could imagine a boxed LLM system for unlearning through prompting, where :

• Only the input and output interfaces are exposed (like ChatGPT);
• Different instances of a powerful LLM are responsible for accurately mimicking different parts of a desired unlearning behavior (for example, one LLM instance specializes in general trivia-style QA while anoother handles sequence completions);
• An orchestrator/router LLM decides which unlearning worker instance to call depending on the input; and
• A composer/summarizer LLM that drafts the final output conforming to the desired unlearning behavior; it may also apply some output filtering.

Some readers may grumble about the heuristic nature of such prompting-based techniques; that there is no proof of unlearning whatsoever. We should keep in mind that fine-tuning based empirical unlearning, as most recent approaches do, is perhaps not fundamentally different. I think it ultimately comes down to the following questions:

• Which of fine-tuning or prompting can better steer model behavior ?
• Which of them are less susceptible to attacks (exposing less surfaces and/or requiring more effort for an adversary to revert the unlearning)?

My intuition of our current models says that both questions point to fine-tuning based unlearning, but this is very much up for debate and can change as we get more powerful models and better defense mechanisms. For example, the recent notion of an instruction hierarchy may help make such as an LLM system less susceptible to malicious prompts.

It might be useful to note that humans don’t really “unlearn” a piece of knowledge either. 5 In fact, by claiming to have unlearned something, we often have: (1) not only learned it well to be able to make the very claim that we have unlearned it, and (2) consciously decided that it’s no longer useful / beneficial to apply this knowledge to our current world state. Who is to say that unlearning for LLMs should be any different?

Unlearning is messy for many reasons. But one of the biggest broken things about unlearning is evaluation. In general, we care about three aspects:

• Efficiency : how fast is the algorithm compared to re-training?
• Model utility : do we harm performance on the retain data or orthogonal tasks?
• Forgetting quality : how much of the “forget data” is actually unlearned? How fast can we recover (re-learn) them?

Evaluating efficiency and model utility are easier; we already measure them during training. The key challenge is in understanding the forgetting quality. 6

If the forget examples are specified, this feels easy too. For example, unlearning a particular image class intuitively means getting a near-chance accuracy on the images in that class. An evaluation protocol may measure accuracy (high on retain & test set, low on forget set) or the likelihood of the forget text sequences (lower the better).

However, these intuitive choices of metrics aren’t necessarily principled or extensible to settings like knowledge unlearning in LLMs. Expecting the model to perform poorly on an unlearned image ignores generalization , as the forget examples could very well be an interpolation/duplicate of certain retain examples. And we don’t always have oracle models that have never seen the forget examples; e.g., do we have LLMs that have never seen New York Times articles?

Evaluating unlearning on LLMs had been more of an art than science. For example, to unlearn “Harry Potter” as an entity, people would visualize how the token probabilities would decay for Harry Potter related text—and some other folks would come along and show that the model can indeed still answer Harry Potter trivia questions. The key issue has been the desperate lack of datasets and benchmarks for unlearning evaluation.

Since 2024, nevertheless, the benchmarking crisis is getting better. There are two recent projects worth highlighting:

• TOFU : A benchmark focusing on unlearning individuals (specifically book authors). It involves asking GPT-4 to create fake author profiles, fine-tuning an LLM on them, and using the fine-tune as the unlearning target model and the original LLM as the oracle “retrained” model. It provides QA pairs on the generated fake authors to evaluate a model’s knowledge of these authors before/after applying unlearning.
• WMDP : A benchmark focusing on unlearning dangerous knowledge, specifically on biosecurity, cybersecurity, and chemical security. It provides 4000+ multiple-choice questions to test a model’s hazardous knowledge before/after applying unlearning. As part of the report the authors also propose an activation steering based empirical unlearning method.

TOFU and WMDP depart from previous unlearning evaluation in that they are both “high-level” and focus on the model’s knowledge retention and understanding as opposed to example-level metrics like forget sequence perplexity. This is particularly relevant for LLMs as they are generally capabale of giving the same answer in many different ways that example-level metrics can’t capture.

Looking forward, I think application-oriented unlearning benchmarks like TOFU and WMDP, as opposed to instance-based evaluation like that of the NeurIPS unlearning challenge , are more useful for evaluating foundation models, owing to the multi-tasking nature of these models and the disparate definitions of “unlearning success” for each of these tasks. Indeed, one might imagine separate benchmarks on unlearning personally identifiable information (PII), copyrighted content, speech toxicity, or even model backdoors . For example, for unlearning PII, we might care about exact token regurgitation, whereas for toxicity, the unlearning metric would be the score reported by a ToxiGen classifier.

4. Practice, pitfalls, and prospects of unlearning

Unlearning is a hard problem, especially in the context of foundation models. As we actively research to make unlearning work in practice, it helps to philosophize a bit on what unlearning really means and whether it is the right solution for our current problems.

Intuitively, unlearning infrequent textual occurrences in LLMs like car accidents in Palo Alto should be easier than unlearning frequent occurrences like “Biden is the US president”, which is in turn easier than unlearning fundamental facts like “the sun rises every day”.

This spectrum of unlearning hardness emerges because as a piece of knowledge becomes more fundamental, it will have more associations with other pieces of knowledge (e.g. as premises or corollaries) and an exponentially larger unlearning scope. In fact, a piece of knowledge can be so embedded in the model’s implicit knowledge graph that it cannot be unlearned without introducing contraditions and harming the model’s utility. 7

This intuition implies that certain unlearning requests are much harder or simply unsatisfiable (any attempts are bound to have flaws). Indeed, humans have experiences that form the basis of their subsequent actions and world models; it is subjective, blurry, and philosophical as to what capacity can humans unlearn their formative past memories.

More broadly, the unlearning hardness problem applies to all kinds of models, and for reasons beyond embeddedness in a knowledge/entailment graph. Let’s consider two more seemingly contradictory intuitions for unlearning hardness:

• An example seen later in the training should be easy to unlearn , since the model would have moved only slightly in weight space (e.g. due to decayed learning rate) and one could either just revert gradients or revert to a previous checkpoint (if stored). In contrast, examples seen early gets “built on” by later examples (in the curriculum learning sense), making them harder to unlearn.
• An example seen later should be harder to unlearn , since examples seen earlier are gradually (or catastrophically) forgotten over the course of training; this may be especially true for LLMs.

Failure to reconcile these intuition would suggest that the interplay across memorization/forgetting , example importance (in the sense of data selection and coresets ), learning hardness (in the sense of prediction flips ), and unlearning hardness is unclear.

Here are some interesting research questions :

• Is there a qualitative/fundamental difference between unlearning “easy” data (e.g. a local news event) and “hard” data (e.g. cats have four legs)?
• If there is a spectrum of unlearning hardness, does there exist a threshold to tell apart what is “easy” and “hard”, and thus what is unlearnable or shouldn’t be unlearned? Does there exist, or can we train, such an oracle classifier? Can humans even tell?
• How does this relate to influence functions and data attribution ? If a certain piece of knowledge (as it manifests in a model’s output) can be attributed to a larger fraction of the training data, does it make it harder to unlearn?
• Can we benchmark how easy is it to unlearn something?

On the surface, unlearning seems to be a promising solution for copyright protection: if a model violates the copyright of some content, we could attempt to unlearn said content. 8 It is conceivable that to resolve copyright violations via unlearning, provable and exact unlearning is necessary (and possibly sufficient); on the other hand, approximate unlearning, without guarantees and with the possibility of being hacked, is certainly insufficient and likely unnecessary.

In practice, however, there is a lot more nuance due to the questionable effectiveness of current unlearning methods and the unclear legal landscape at the intersection of AI and copyright. Since I am no legal expert (and clearly none of this section constitutes legal advice), we will mostly focus on asking questions. The central question seems to be: is unlearning the right solution for copyright protection?

Recall that the fair use doctrine 9 permits limited use of copyrighted material contigent on four factors: (1) purpose and character of the use (“transformativeness”), (2) the nature of the copyrighted work, (3) amount and substantiality of the use, and (4) the effect on material’s value. If the use of copyrighted content in a model qualifies as fair use, then unlearning such content from the model is unnecessary.

Suppose a model is trained on some copyrighted content and is risking copyright violation, as in New York Times v. OpenAI . Should OpenAI invest in (empirical) unlearning algorithms on ChatGPT? Or should they focus on the transformativeness axis of fair use and invest in deploying empirical guardrails , such as prompting, content moderation, and custom alignment to prevent the model from regurgitating training data? The latter seems to be what’s being implemented in practice.

More broadly, there could also be economic solutions to copyright violation as alternatives to unlearning. For example, model owners may provide an exact unlearning service (e.g. via periodic retraining) while also offering to indemnify model users for copyright infringement in the mean time, as seen in the case of OpenAI’s “ Copyright Shield ”. People are also starting to explore how one may price copyrighted data using Shapley values. In general, it is unclear right now how much of a role (if any) unlearning will play for resolving copyright related issues. Exact unlearning (extending to retrieval-based systems, see next section) does hold promises since deletion is clean and provable, but it seems that legally binding auditing procedures/mechanisms need to be first in place.

An obvious alternative to unlearning is to not learn at all. One way this could manifest for an LLM is that we take all content from the pre-training set that may receive unlearning requests (e.g., New York Times articles) and put them to an external data/vector store. Any questions relating to them will then be RAG ’ed during inference, and any unlearning requests can be trivially satisfied by removing the data from the database. Min et al. demonstrates that this approach can be competitive to (though not quite matching) the trained baseline in terms of final perplexity.

Retrieval-based solutions are promising because of the increasing capabilities of the base models to reason in-context. However, there are few considerations before taking retrieval systems as the no-brainer solution to unlearning:

• Removing protected content from pre-training corpus can be a hard de-duplication problem. Much like removing data contamination is hard , how can we be sure that paraphrases, quotations/citations, or other adaptations of the protected content are removed?
• What if the data to be unlearned can’t be retrieved? Today we fine-tune many things into a model that aren’t documents or knowledge items; for example, it is unclear (yet) if things like as human preferences and desired behaviors (e.g. ability to write concisely) can be “retrieved” from a database.
• Dumping stuff in-context can open new attack surfaces. Many RAG methods for LLMs work by putting related content in-context and ask the model to reason on them. Having the protected data in-context means they are now more susceptible to data extraction (simple prompting attacks may work just fine ).
• Utility gap between retrieval and training. While there is evidence that retrieval-based solutions can be competitive, there is no general consensus that retrieval alone can replace fine-tune workloads; indeed, they can be complementary . More broadly, what if the space of unlearnable data is too large such that if all of it goes to an external store, the base model wouldn’t be as useful?

As models become more capable and are granted agency , one concrete application domain for unlearning that is gaining traction is AI safety .

Roughly speaking, safety concerns stem from a model’s knowledge (e.g., recipe of napalm ), behaviors (e.g., exhibiting bias ), and capabilities (e.g., hacking websites). Examining current AI systems and extrapolating forward, one may imagine the following examples to apply unlearning and improve AI safety:

• removing hazardous knowledge , as seen in the WMDP benchmark;
• removing model poisons and backdoors , where models respond to adversarially planted input triggers;
• removing manipulative behaviors , such as the ability to perform unethical persuasions or deception;
• removing bias and toxicity ; or even
• removing power-seeking tendencies .

For safety-oriented applications, it is worth noting that unlearning should be treated as a post-training risk mitigation and defense mechanism , alongside existing tools like alignment fine-tuning and content filters. And as with any tool, we should view unlearning through its trade-offs in comparison to other tools in the toolbox (e.g., unlearning is more adaptive but more expensive than content filters), as opposed to brushing it off because of the potential lack of guarantees and efficacy.

Acknowledgements : The author would like to thank Aryaman Arora, Jiaao Chen, Irena Gao, John Hewitt, Shengyuan Hu, Peter Kairouz, Sanmi Koyejo, Xiang Lisa Li, Percy Liang, Eric Mitchell, Rylan Schaeffer, Yijia Shao, Chenglei Si, Pratiksha Thaker, Xindi Wu for helpful discussions and feedback before and during the drafting of this post. Any hot/bad takes are those of the author.

If you find this post helpful, it can be cited as:

Liu, Ken Ziyu. (Apr 2024). Machine Unlearning in 2024. Ken Ziyu Liu - Stanford Computer Science. https://ai.stanford.edu/~kzliu/blog/unlearning .

Technically, SISA may not give exact unlearning in the sense of identical model distributions between the retrained model and the unlearned model, since after a sequence of unlearning requests, the data shards may end up in a state that we wouldn’t otherwise get into in the first place (e.g., some shards have way more data than others after unlearning). For practical purposes, nevertheless, this is subtle enough that the nice properties about exact unlearning, as discussed later in the section, would still hold.  ↩

It is also worth noting that the unlearning metric used in the NeurIPS unlearning challenge was disputed: why should we stick to a DP-like distributional closeness metric to a single secretly-kept re-trained model, when retraining itself can give a different model due to randomness?  ↩

More broadly, “unlearning” falls under the umbrella of “model editing” in the sense that a deletion is also an edit. Similarly, one could argue that the concept of “continual learning” falls under the umbrella too, where an association (say an input/label pair, or a piece of factual association) is updated by deleting of an old association and creating a new, clearly specified association. One could imagine using continual learning to help achieve unlearning and vice versa.  ↩

There is also evidence that in-context demonstrations mostly serve to elicit a particular behavior and that the labels don’t even matter that much. It’s unclear yet how we could reconcile this finding with “in-context unlearning”.  ↩

Humans do forget things though, which is different. The ML analogy might be “catastrophic forgetting”; humans similarly forget things under information overload.  ↩

In particular, recall that for exact unlearning , understanding forgetting quality isn’t strictly necessary because the algorithm would remove the forget data from the picture by construction (through retraining). Thus it may be acceptable even if the unlearned model does well on the forget set (as it could be a result of generalization from the retain set). We will focus the discussions of unlearning evaluation on approximate unlearning.  ↩

Note that this “embeddedness” of a piece of data is related but distinct from whether the data is in or out of distribution , which should also affects how an unlearning algorithm should behave (e.g. unlearning a perfect inlier should be no-op for an ideal unlearning algorithm).  ↩

Of course, we must first verify that such content has been trained on by the model in the first place. We can be almost certain that contents like Wikipedia articles are trained on, but we are generally less sure about a random blogpost somewhere on the internet. This is basically the membership inference problem.  ↩

Fair use is a doctrine applicable specifically in the United States. The reader should refer to related doctrines in corresponding jurisdictions, such as fair dealings in Commonwealth countries.  ↩