literature review funnel

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

3 Planning Your Research: Reviewing the Literature and Developing Questions

ESSENTIAL QUESTIONS

• What is relevant literature? What are the best ways to find it?
• What are the best ways to organize your relevant literature?
• What are the intended outcomes of reviewing your relevant literature?

Nearly all research begins with a review of literature that is relevant to the topic of research, even if it is only a casual review. Reviewing the available literature on your topic is a vital step in the research process. The literature review process provides an anchor for your inquiry. O’Leary (2004, p. 66) states, the “production of new knowledge is fundamentally dependent on past knowledge” because “it is virtually impossible for researchers to add to a body of literature if they are not conversant with it.” By reviewing the literature in the initial stages of the inquiry process, researchers are better able to:

• Understand their topic;
• Develop and focus a topic;
• Provide a clear rationale for, or better situate, their topic;
• Fine-tune their research questions.

In terms of thinking about methodology and the actual research process, reviewing the literature can help researchers:

• Identify well-vetted data collection and analysis methods on their topic;
• Determine whether to replicate a previous study, or develop a completely new study;
• Add rigor and validity to the research by validating the topic, methods, and significance.

Lastly, reviewing the literature also helps the researcher make sense of their findings, in both their field of study and in their educational context, by:

• Assessing whether the findings correlate with findings from another study;
• Determining which of the findings are different than previous studies;
• Determining which of the findings are unique to the researcher’s educational context. [1]

As you may see, the literature review is the backbone, anchor, or foundation of your research study. Overall the review of literature helps you answer three important questions that are the result of the bullet points outlined above. The literature review helps you answer the following:

• What do we know about your topic?
• What do we not know about your topic?
• How does your research address the gap between what we know and what we don’t about your topic?

After reviewing the literature, if you are able to answer those three questions, you will have a very clear and well-rationalized justification for your inquiry. If you cannot answer those questions, then you should probably keep reviewing the literature by looking for related topics or synonyms of major concepts.

While an extensive review of the literature about your topic of study is expected, you should also be realistic as to what you are able to manage. For topics that have a lot of research literature available, make sure you establish parameters for your research, such as:

• Temporal (e.g., only articles in the last 5 or 10 years)
• Content Area (e.g., only in science and math classrooms)
• Age or grade (e.g., only middle school classrooms)
• Research Subject (e.g., girls only, teachers, struggling readers)

These categories provide only a few examples, but parameters like these can make your review of literature much more manageable and your study much more focused.

What Types of Literature Should You Consider in Your Review?

It is helpful to consider the characteristics, purposes, and outcomes of different types of literature. Below are four broad categories I identify within educational research literature. I want to emphasize that my categories are in no way definitive, and only represent my own understanding.

Policy-Based Literature

Policy-based literature includes official documents that outline education policy with which the practitioner needs to be familiar. For example, the Common Core Standards or Content Standards are often refenced in articles to situate the need for research in relation to the standards; if my topic was on place-based education with middle school social studies students, I might have to look at national social studies standards. There also may be initiatives launched by organizations or researchers that become accepted practice. The documents that launch these initiatives (e.g., reports, articles, speeches) would also be useful to review. An example would be the Report of the National Reading Panel: Teaching Children to Read report from the National Reading Panel. These documents may provide rationale, based on the theories and concepts they utilize, and they may provide new ways of thinking about your topic. Similarly, if your topic is based on the local context, recent newspaper articles could also provide policy-type insights. All of these policy-based insights will be useful in providing the landscape or background for your work.

Theoretical Literature

Once you have identified your theoretical perspective, it is also important to locate your research within the appropriate theoretical literature. Many of you may be engaged in highly practice-based or small-scale research and wonder if you need a theoretical basis in your literature review. Regardless of the extent of your project, theoretical literature will help with the rigor and validity of your study and will help identify any theoretical views that underlie your topic. For example, if your study focuses on the place-based education in enhancing social studies students’ learning, it is highly probable that you would cite Kolb’s (1984/2014) work on Experiential Learning. By using Kolb’s work, you situate your research theoretically in the area of experiential learning.

Applicable Literature

Applicable literature will account for the bulk of your literature review. The previous two types of literature provide indication as to where your research is rationalized professionally and situated theoretically. Applicable literature will mainly come from journals related to your specific field of study. If I was doing a study in a social studies classroom, I would look at the journals The Social Studies, Social Education, and Social Studies Research and Practice . Use Google Scholar or your university library databases to examine literature in your specific area. When using these search engines and databases, start as specific as possible with your topic and related concepts. Using the example of place-based learning from above, I would search for “place-based learning” and “social studies” and “middle school” and “historic sites”. If I did not find many articles with this first search, then I would remove “historic sites” and search again. Books or handbooks on research may also have some useful studies to support your literature review section.

Methodological Literature

When sharing or reporting your work, you will want to review and cite research methodology literature to justify the methods you chose. When reading other research articles, pay attention to the research methods used by researchers. It is especially important to find articles that use and cite action research methodology. This type of literature will provide further support of your data gathering and analysis methods. Again, your methods should fit within your theoretical and epistemological stances. In addition, you’ll want to review data collection methods and potentially borrow or adapt rubrics or surveys from other studies.

Sources of Relevant Literature

When searching for these four types of literature, there are two ways to think about possible sources:

• primary sources include government publications, policy documents, research papers, dissertations, conference presentations and institutional occasional papers with accounts of research;
• secondary sources use primary sources as references, such as papers written for professional conferences and journals, books written for practicing professionals and book reviews. This is often called “reference mining” as you look through the reference lists of other studies and then return to the primary source that was cited.

Secondary sources are often just as valuable as primary sources, or potentially more valuable. When beginning your search, secondary sources can provide links to a wealth of primary sources that the secondary source author has already vetted for you, and likely with similar intentions. This is especially true of research handbooks. You will come across both types of literature wherever you search, and they both provide a landscape for your topic and add value to your literature review.

Regardless of being a primary or secondary source, you want to make sure the literature you review is peer-reviewed. Peer-reviewed simply means that the article was reviewed by two to three scholars in the field before it was published. Books, or edited books, would have also gone through a peer-review process. We often recommend teachers to look at professional books from reputable publishing companies and professional organizations, such as ASCD, NCTE, NCTM, or NCSS. This is a way for scholars to objectively review each other’s work to maintain a high level of quality and ethics in the publication of research. Most databases have mostly peer-reviewed journals, and often provide a filter to sort out the non-peer-reviewed journals.

Using the Internet

The internet is a valuable research tool and is becoming increasingly efficient and reliable in providing peer-reviewed literature. Sites like Google Scholar are especially useful. Often, and depending on the topic, the downside of internet-based searches is that it will generate thousands or millions of sources. This can be overwhelming, especially for new researchers, and you will have to develop ways to narrow down the results.

Professional organization websites will also have resources or links to sources that have typically been vetted. With all internet sources, you should evaluate the information for credibility and authority.

Evaluating sources from the Internet

Evaluating internet sources is a whole field of study and research within itself, and an in-depth discussion would take away from the focus of this book. However, O’ Dochartaigh (2007) provides a chapter to help guide the internet source evaluation process. Here is a brief summary, based on O’ Dochartaigh’s book, to give you a general idea of the task of evaluating sources:

• Examine if the material belongs to an advocacy group. Many times, these sources are fine, however, they require extra examination for bias or funding interests.
• As mentioned above, many academic papers are published in refereed journals which are subject to peer-review. Papers found on academic or university websites are typically refereed in some manner; however, some papers are posted by academics on their personal sites and have not been reviewed by other academics. Papers published solely by academics or other experts require further scrutiny before citing.
• When you are reviewing newspaper and magazine articles from the internet be weary of potential conflicts of interest based on the political stance of that periodical.

Therefore, it is wise to consider the objectivity of any source you find on the internet before you accept the literature.

We always recommend that students consider a few questions in their evaluation of sources, which are similar to the formal questions outlined by O’ Dochartaigh (2007):

• Is it clear who is responsible for the document?
• Is there any information about the person or organization responsible for the page?
• Is there a copyright statement?
• Does it have other publications that reinforce its authority?
• Are the sources clearly listed so they can be verified?
• Is there an editorial involvement?
• Are the spelling and grammar correct?
• Are biases and affiliations clearly stated?
• Are there dates for when the document was last updated or revised?

Organizing your Literature

When you begin, here are some things to think about as your read the literature. Again, these are not definitive, but merely provided for guidance. These questions are especially focused on other action research literature:

Questions to Think about as You Examine the Literature

• What was the context of their research?
• Who was involved? Was it a collaborative project?
• Was the choice of using action research as a method justified? Are any models discussed?
• What ‘actions’ actually took place?
• How was data gathered?
• How was data analyzed?
• Were ethical considerations addressed? How?
• What were the conclusions? Were they justified using appropriate evidence?
• Was the report accessible? Useful?
• Is it possible to replicate the study?

Regardless of the amount of literature you review, your challenge will be to organize the literature in way that is manageable and easy to reference. It is important to keep a record of what you read and how it relates conceptually to your topic. Some researchers even use the questions above to organize their literature. It is easy to read and think about the content of an article by making brief notes, however, this is often not enough to initially begin to develop your study or write about your findings. I will state the obvious here: organizing your literature search efficiently from the start is vital!

No matter how you choose to record or document the articles you read (e.g., paper, computer, photo), I would suggest thinking about the format in terms of index cards. Index cards are a very practical and simple model because the space limits you to be precise in recording vital information about each article. I typically create a document on my computer, allow each article the space of an index card, and focus on recording the following information:

• Journal/Book Chapter Title
• Main Arguments/Key findings
• Pertinent Quote(s)
• Implications
• Connective Points (how does it relate to my work and/or other articles)

I find that these aspects provide the information I need to be refreshed on the article and to be able to use it upon review.

There are also a lot of computer applications that are very useful and efficient in managing your literature. For example, Mendeley © provides comprehensive support for reviewing literature, even allowing you to store the article itself and make comments or highlights in text. There are also many citation apps that are helpful if you continue this research agenda and use roughly the same literature for each project.

Using the Literature

Think ahead to when you have collected and read a good amount of literature on your topic. You are now ready to use the literature to think about your topic, your research question, and the methods you plan to use. It might be helpful to peek ahead to Chapter 7 where I discuss writing the literature review for a report to give you an idea of the end goal. The primary purpose of engaging in a literature review is to provide knowledge to construct a framework for understanding the landscape of your topic. I often suggest for students to think of it as constructing an argument for your research decisions, or as if you are telling a story of how we got to this point in researching your topic. Either way you are situating your research in what we know and don’t know about your topic.

Naturally we tend to think about, and potentially write about, the literature in relation to the article’s author (e.g. Clark and Porath (2016) found that…). However, more commonly today in educational research you will find that literature reviews are organized by themes. It can be a little more organic to think about literature in terms of themes because they emerge or become more defined as you read. Also thinking thematically allows the articles to naturally connect and build on each other, whereas thinking in terms of authors can fragment thinking about the topic. In terms of thinking thematically, here are some guidelines:

• Identify the significant themes that have emerged organically from the literature review. These themes would be concepts or ideas that you typed or wrote down in your note-taking or management system.
• Introduce the common concepts or ideas by themes, instead of by authors’ disjointed viewpoints. Paragraphs in a thematic literature review begin like: The research on teacher self-efficacy has identified several key factors that contribute to strong self-efficacy… .
• Lastly, once you have introduced each theme and explained it, then present evidence from your readings to demonstrate the parameters of the knowledge on the theme, including areas of agreement and disagreement among researchers. Using the evidence, explain what the evidence for the theme means to your topic and any of your own relational or critical commentary.

Another way to think about structuring a literature review is a funnel model. A funnel model goes from broad topic, to sub-topics, to link to the study being undertaken. You can think of a literature review as a broad argument using mini-arguments. To use the funnel model, list your topic and the related subtopics, then design questions to answer with the literature. For example if our topic was discussion-based online learning, we might ask the following questions before reading the literature:

• Why is discussion important in learning?
• How does discussion support the development of social, cognitive, and teacher presence in an online course?
• What does research say about the use of traditional discussion boards ?
• What does the research say about asynchronous, video-based discussion ?
• How have other researchers compared written and video responses?
• How does this literature review link to my study?

O’Leary (2004) provides an interesting representation and model of the purpose for the literature review in the research process, in Figure 3.1. We will leave this for you to think about before moving on to Chapter 4.

• We will talk about this aspect of literature reviews further in Chapters 6 and 7. ↵

Action Research Copyright © by J. Spencer Clark; Suzanne Porath; Julie Thiele; and Morgan Jobe is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

Share This Book

How To Structure Your Literature Review

3 options to help structure your chapter.

By: Amy Rommelspacher (PhD) | Reviewer: Dr Eunice Rautenbach | November 2020 (Updated May 2023)

Writing the literature review chapter can seem pretty daunting when you’re piecing together your dissertation or thesis. As  we’ve discussed before , a good literature review needs to achieve a few very important objectives – it should:

• Demonstrate your knowledge of the research topic
• Identify the gaps in the literature and show how your research links to these
• Provide the foundation for your conceptual framework (if you have one)
• Inform your own  methodology and research design

To achieve this, your literature review needs a well-thought-out structure . Get the structure of your literature review chapter wrong and you’ll struggle to achieve these objectives. Don’t worry though – in this post, we’ll look at how to structure your literature review for maximum impact (and marks!).

But wait – is this the right time?

Deciding on the structure of your literature review should come towards the end of the literature review process – after you have collected and digested the literature, but before you start writing the chapter. 

In other words, you need to first develop a rich understanding of the literature before you even attempt to map out a structure. There’s no use trying to develop a structure before you’ve fully wrapped your head around the existing research.

Equally importantly, you need to have a structure in place before you start writing , or your literature review will most likely end up a rambling, disjointed mess. 

Importantly, don’t feel that once you’ve defined a structure you can’t iterate on it. It’s perfectly natural to adjust as you engage in the writing process. As we’ve discussed before , writing is a way of developing your thinking, so it’s quite common for your thinking to change – and therefore, for your chapter structure to change – as you write. 

Need a helping hand?

Like any other chapter in your thesis or dissertation, your literature review needs to have a clear, logical structure. At a minimum, it should have three essential components – an  introduction , a  body   and a  conclusion . 

Let’s take a closer look at each of these.

1: The Introduction Section

Just like any good introduction, the introduction section of your literature review should introduce the purpose and layout (organisation) of the chapter. In other words, your introduction needs to give the reader a taste of what’s to come, and how you’re going to lay that out. Essentially, you should provide the reader with a high-level roadmap of your chapter to give them a taste of the journey that lies ahead.

Here’s an example of the layout visualised in a literature review introduction:

Your introduction should also outline your topic (including any tricky terminology or jargon) and provide an explanation of the scope of your literature review – in other words, what you  will   and  won’t   be covering (the delimitations ). This helps ringfence your review and achieve a clear focus . The clearer and narrower your focus, the deeper you can dive into the topic (which is typically where the magic lies). 

Depending on the nature of your project, you could also present your stance or point of view at this stage. In other words, after grappling with the literature you’ll have an opinion about what the trends and concerns are in the field as well as what’s lacking. The introduction section can then present these ideas so that it is clear to examiners that you’re aware of how your research connects with existing knowledge .

2: The Body Section

The body of your literature review is the centre of your work. This is where you’ll present, analyse, evaluate and synthesise the existing research. In other words, this is where you’re going to earn (or lose) the most marks. Therefore, it’s important to carefully think about how you will organise your discussion to present it in a clear way. 

The body of your literature review should do just as the description of this chapter suggests. It should “review” the literature – in other words, identify, analyse, and synthesise it. So, when thinking about structuring your literature review, you need to think about which structural approach will provide the best “review” for your specific type of research and objectives (we’ll get to this shortly).

There are (broadly speaking)  three options  for organising your literature review.

Option 1: Chronological (according to date)

Organising the literature chronologically is one of the simplest ways to structure your literature review. You start with what was published first and work your way through the literature until you reach the work published most recently. Pretty straightforward.

The benefit of this option is that it makes it easy to discuss the developments and debates in the field as they emerged over time. Organising your literature chronologically also allows you to highlight how specific articles or pieces of work might have changed the course of the field – in other words, which research has had the most impact . Therefore, this approach is very useful when your research is aimed at understanding how the topic has unfolded over time and is often used by scholars in the field of history. That said, this approach can be utilised by anyone that wants to explore change over time .

For example , if a student of politics is investigating how the understanding of democracy has evolved over time, they could use the chronological approach to provide a narrative that demonstrates how this understanding has changed through the ages.

Here are some questions you can ask yourself to help you structure your literature review chronologically.

• What is the earliest literature published relating to this topic?
• How has the field changed over time? Why?
• What are the most recent discoveries/theories?

In some ways, chronology plays a part whichever way you decide to structure your literature review, because you will always, to a certain extent, be analysing how the literature has developed. However, with the chronological approach, the emphasis is very firmly on how the discussion has evolved over time , as opposed to how all the literature links together (which we’ll discuss next ).

Option 2: Thematic (grouped by theme)

The thematic approach to structuring a literature review means organising your literature by theme or category – for example, by independent variables (i.e. factors that have an impact on a specific outcome).

As you’ve been collecting and synthesising literature , you’ll likely have started seeing some themes or patterns emerging. You can then use these themes or patterns as a structure for your body discussion. The thematic approach is the most common approach and is useful for structuring literature reviews in most fields.

For example, if you were researching which factors contributed towards people trusting an organisation, you might find themes such as consumers’ perceptions of an organisation’s competence, benevolence and integrity. Structuring your literature review thematically would mean structuring your literature review’s body section to discuss each of these themes, one section at a time.

Here are some questions to ask yourself when structuring your literature review by themes:

• Are there any patterns that have come to light in the literature?
• What are the central themes and categories used by the researchers?
• Do I have enough evidence of these themes?

PS – you can see an example of a thematically structured literature review in our literature review sample walkthrough video here.

Option 3: Methodological

The methodological option is a way of structuring your literature review by the research methodologies used . In other words, organising your discussion based on the angle from which each piece of research was approached – for example, qualitative , quantitative or mixed  methodologies.

Structuring your literature review by methodology can be useful if you are drawing research from a variety of disciplines and are critiquing different methodologies. The point of this approach is to question  how  existing research has been conducted, as opposed to  what  the conclusions and/or findings the research were.

For example, a sociologist might centre their research around critiquing specific fieldwork practices. Their literature review will then be a summary of the fieldwork methodologies used by different studies.

Here are some questions you can ask yourself when structuring your literature review according to methodology:

• Which methodologies have been utilised in this field?
• Which methodology is the most popular (and why)?
• What are the strengths and weaknesses of the various methodologies?
• How can the existing methodologies inform my own methodology?

3: The Conclusion Section

Once you’ve completed the body section of your literature review using one of the structural approaches we discussed above, you’ll need to “wrap up” your literature review and pull all the pieces together to set the direction for the rest of your dissertation or thesis.

The conclusion is where you’ll present the key findings of your literature review. In this section, you should emphasise the research that is especially important to your research questions and highlight the gaps that exist in the literature. Based on this, you need to make it clear what you will add to the literature – in other words, justify your own research by showing how it will help fill one or more of the gaps you just identified.

Last but not least, if it’s your intention to develop a conceptual framework for your dissertation or thesis, the conclusion section is a good place to present this.

Example: Thematically Structured Review

In the video below, we unpack a literature review chapter so that you can see an example of a thematically structure review in practice.

Let’s Recap

In this article, we’ve  discussed how to structure your literature review for maximum impact. Here’s a quick recap of what  you need to keep in mind when deciding on your literature review structure:

• Just like other chapters, your literature review needs a clear introduction , body and conclusion .
• The introduction section should provide an overview of what you will discuss in your literature review.
• The body section of your literature review can be organised by chronology , theme or methodology . The right structural approach depends on what you’re trying to achieve with your research.
• The conclusion section should draw together the key findings of your literature review and link them to your research questions.

If you’re ready to get started, be sure to download our free literature review template to fast-track your chapter outline.

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

27 Comments

Great work. This is exactly what I was looking for and helps a lot together with your previous post on literature review. One last thing is missing: a link to a great literature chapter of an journal article (maybe with comments of the different sections in this review chapter). Do you know any great literature review chapters?

I agree with you Marin… A great piece

I agree with Marin. This would be quite helpful if you annotate a nicely structured literature from previously published research articles.

Awesome article for my research.

I thank you immensely for this wonderful guide

It is indeed thought and supportive work for the futurist researcher and students

Very educative and good time to get guide. Thank you

Great work, very insightful. Thank you.

Thanks for this wonderful presentation. My question is that do I put all the variables into a single conceptual framework or each hypothesis will have it own conceptual framework?

Thank you very much, very helpful

This is very educative and precise . Thank you very much for dropping this kind of write up .

Pheeww, so damn helpful, thank you for this informative piece.

I’m doing a research project topic ; stool analysis for parasitic worm (enteric) worm, how do I structure it, thanks.

comprehensive explanation. Help us by pasting the URL of some good “literature review” for better understanding.

great piece. thanks for the awesome explanation. it is really worth sharing. I have a little question, if anyone can help me out, which of the options in the body of literature can be best fit if you are writing an architectural thesis that deals with design?

I am doing a research on nanofluids how can l structure it?

Beautifully clear.nThank you!

Lucid! Thankyou!

Brilliant work, well understood, many thanks

I like how this was so clear with simple language 😊😊 thank you so much 😊 for these information 😊

Insightful. I was struggling to come up with a sensible literature review but this has been really helpful. Thank you!

You have given thought-provoking information about the review of the literature.

Thank you. It has made my own research better and to impart your work to students I teach

I learnt a lot from this teaching. It’s a great piece.

I am doing research on EFL teacher motivation for his/her job. How Can I structure it? Is there any detailed template, additional to this?

You are so cool! I do not think I’ve read through something like this before. So nice to find somebody with some genuine thoughts on this issue. Seriously.. thank you for starting this up. This site is one thing that is required on the internet, someone with a little originality!

I’m asked to do conceptual, theoretical and empirical literature, and i just don’t know how to structure it

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Literature reviews

• Book a session

What is a literature review?

Finding your sources, structuring your review, critical writing, literature reviews, critiques and annotated bibliographies.

• Science and Health-based reviews This link opens in a new window
• Quick resources (5-10 mins)
• e-learning and books (30 mins+)
• SkillsCheck This link opens in a new window
• ⬅ Back to Skills Centre This link opens in a new window

Looking for sessions and tutorials on this topic? Find out more about our session types and how to register to book for sessions. You can view our full and up-to-date availability in UniHub  Appointments and Events . 

Not sure where to start developing your academic skills? Take the  SkillsCheck for personalised recommendations on how to build your academic writing and study skills alongside your course.

Literature reviews take on many forms at university: you could be asked to write a literature review as a stand-alone document or as part of a dissertation or thesis. You may also be asked to write an annotated bibliography or a critical review - both of these assignments are closely related to literature reviews, and follow many of the same conventions.

A literature review is an extended piece of writing that should collate, link and evaluate key sources related to a chosen topic or research question. Rather than simply summarising the existing research on your chosen topic, you should aim to show which papers can be clustered around a similar theme or topic - they may have a shared methodology, or have been carried out in the same context. You will be looking for strengths and weaknesses in the research, questioning the relevance and significance of the results in relation to your topic, and looking for any gaps or under researched areas. Your writing should make these thoughts and evaluations clear to the reader, so that they have a good understanding and overview of the body of research you have chosen to investigate.

Here is a short video that explains a literature review from the perspective of the reader:  

Salter, J. [Dr. Jodie Salter]. (2016, March 14). Writing the Literature Review: A Banquet Hall Analogy [Video file]. Retrieved from: https://www.youtube.com/watch?v=QE_Us8UjS6

Using Library Search

The more you read around your subject, the more familiar you will become with the current literature, and you will start to build a map of the sources you already have, and the information you are missing.  A clear search strategy can help fill these gaps in your knowledge, and your themes or topics of interest can be used as key search terms when looking for further resources.   

Top tips for searching the Library Gateway

• Use Boolean terms to help the search engine recognise which words should be treated as a phrase. For example, if you search “costume design” , the search engine will know to treat “costume design” as a phrase, not two separate words.  
• You can then add AND and OR to add in additional terms and synonyms. For example, “costume design” AND “film” will only find articles or sources that include both of these terms together, helping you to narrow your search. To go wider, think about adding in synonyms using the OR function: “costume design” AND “film” or “cinema” or “movies”.  
• You can also use an asterisk (*) to search for a word stem to help widen your search. For example, if you search teach*, this will find articles that include the word teach, teacher, teachers, teaching and so on.  
• Decide at the start on what your inclusion and exclusion criteria  will be. These might include limits on: - Date of publication - Language - Type/group of participants - Peer-reviewed journals - Keywords and synonyms - Type of study, ie. systematic review, case study etc.  

Search strategies

Explore the following resources for more information on search strategy models:

• PICO (Population, Intervention, Comparison, Outcome) - Used primarily in Health Sciences but offers a clear step-by-step approach to literature searching that could be adapted for use in other subjects.
• SPIDER - Developed from the PICK method, SPIDER searching is used mainly in qualitative research to identify a phenomenon or behaviour, rather than a specific intervention (more quantitative).

Finding the gap

You might have heard about  finding the gap  in your literature review - but what does this mean?

Looking for the gap in the literature means finding an aspect of your topic that hasn't been fully explored by researchers. This might be because you are researching a new technique or technology, or that your method or approach hasn't been used before in your field of study. You don't always need to find a gap, but it is a good way of demonstrating your literature searching skills and ability to compare a wide range of different sources. If you are able to find one, introduce the gap towards the end of the literature review, so that the reader can trace your path through the evidence first.

Literature review structure: A three-tier model

Imagine you are explaining your dissertation topic to a friend for the first time. Even for someone on the same degree course, they would need some context on the topic before you introduced more detail and complex examples.

A literature review follows the same  ‘funnel’ narrative , moving from general themes to more specific detail:

• Appraisal grid A template for taking notes from reading that enables you to find clear links and similarities for discussion in your literature review.
• PEP tables A template for organising notes from your reading by themes, theories and perspectives.

Paragraph structure

Each paragraph of your literature review should bring together or synthesise two or more pieces of reading (these could be articles, book chapters, reports, videos, policy documents etc.) 

Synthesis is the term we use in academic writing to describe the process of creating an opinion or argument based on a trend you find in the literature. If you are able to synthesis evidence, you are not only creating a robust argument (by avoiding relying too heavily on just one piece of writing) but you are also showing that you are a critical writer that can make conclusions based on a diverse range of evidence. Bingo!

As in other forms of academic writing, the paragraphs in your literature review should have four key sections:

Compare the following paragraphs against this four-part structure - which version is more critical?  

Although both paragraphs use the TIED structure, we can see that the discussion in paragraph B is much more developed, and gives a specific suggestion about how future research could be conducted. We can also see that the evidence in paragraph B is clearly linked together , and that the conclusions or critical features of the papers are explained to the reader. Although drawing on the same evidence, paragraph A summarises and describes the research papers , rather than giving an evaluation or clear comparison of the different sources.

Focusing on the discussion sections (in bold), we can see that paragraph B is more critical , as it answers a key questions to keep in mind when writing critically: 'so what?'  What conclusion or take home message do you want the reader to get from the evidence you have presented?  ‘Therefore’, ‘Consequently’ and ‘As a result’ are all good terms to use here, as they prompt you to be clear and explicitly explain on interpretation of the source you have included.

Other  types of literature review

Not all literature reviews form part of a dissertation. Use the tabs below for guidance on different assignment formats related to literature reviews:

• What are the emotional and behavioral impacts of therapy dogs for autistic children?
• How might aptitude be tested and measured in puppies selected for guide dog training?
• What are the key success factors for dogs as social media influencers on Instagram and Facebook?

Your initial reading will help you to identify trends or themes in the literature that might help to focus your search. You can then follow the standard structure for writing a  literature review, using the  funnel structure  from this guide.

A critical review, or ‘critique’, involves breaking a journal article down into its key sections so that you can evaluate the strengths and weaknesses of each part. Making notes on each of the following headings is a useful way to kickstart your analysis of any article:

• Research aim
• Research approach (ie. quantitative)
• Ethical issues
• Data collection method
• Data analysis method
• Generalisability/transferability

This list is not exhaustive and depending on your discipline, there may be other relevant categories to focus on in the article, such as theoretical models or implications for practice. The subheadings from the article will also provide an overview of the key sections to include in your review, and you may already have an idea from your wider reading of what sections often appear in articles in your field of study. Breaking down the article in this way allows you to focus your critique and evaluation, highlighting significant or relevant aspects of the article to the reader . Your assessment criteria will help you to identify which elements of the article to include in your critique: for example, if you needed to include a reflection on how the article links to your professional practice, it would make sense to include your thoughts on the articles key findings and transferability in your critique. For examples of sentence starters and academic language to use in your critical review, take a look at the following resources:

• Writing a critical review, UCL
• Academic Phrasebank, University of Manchester

An annotated bibliography combines a correctly formatted list of references (APA) with a short paragraph that gives:

• a short summary of the source, that picks out the key points of the article, such as context and setting, participants and conclusions;
• a brief evaluation of the strengths and weaknesses of the article;
• a sentence or two on the relevance of the source to your question or topic – what does it contribute to your knowledge of the subject, and in what ways might its relevance be limited?

​ ​ Sources are not discussed together in the same paragraph, but the document itself will have a key theme or topic that ties the different sources together – almost like a module reading list: Brym, R., Godbout, M., Hoffbauer, A., Menard, G. & Huiquan Zhang, T. (2014) Social media in the 2011 Egyptian uprising. The British Journal of Sociology , 65 : 266-271. https://doi.org/10.1111/1468-4446.12080 This article conducts a comparative analysis of quantitative data on social media usage and political engagement during the 2011 Egyptian uprising, using new bit.ly and Gallup survey results. The study generates a large amount of data on the key differences in social media usage between active demonstrators and sympathetic onlookers. Most significantly, the study explores the key drivers of participating in social unrest, such as a lack of confidence in the government, and how these are facilitated by social media. However, by only gathering quantitative data, the study is limited in its ability to provide an insight into how protestors narrate and explain their involvement in the protests in their own words. Overall, this article offers significant evidence to support a study of the importance of social media in contemporary political movements, and is particularly useful as one of few studies to focus on events outside of Europe and North America. Be sure to check your assessment criteria for tips on how you should evaluate your sources: for example, you might be asked to include specific methodology types or to link your sources to professional practice.

Two key principles apply to every literature review, whether it is part of a dissertation or an individual assignment:

 1. A literature review is more than just a list of sources. The articles and evidence you include must be linked together around shared themes and characteristics, or highlight significant disagreements and contrast. Map your reading using keywords or themes that occur in multiple articles - these can be used as subheadings in your draft literature review.

2. While it is important to show that you are familiar with research in your field, you also need to show that you can evaluate and offer interpretations of the evidence you present to the reader. Remember to keep answering the 'so what?' question as you write.  

• << Previous: Book a session
• Next: Science and Health-based reviews >>
• Last Updated: Apr 17, 2024 1:52 PM
• URL: https://libguides.shu.ac.uk/literaturereviews

Funnel Plot in a Systematic Review

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

Systematic reviews are considered the gold standard in evidence-based research. These intensive “studies of studies” involve taking a systematic approach to collect, assess, and synthesize relevant literature on a specific subject, including peer-reviewed journal articles, gray literature, and other sources, to answer a well-defined research question. Systematic reviews, like other reviews, can also be prone to bias. The bias types that are most evaluated in systematic reviews are selection bias, reporting bias, detection bias, and attrition bias. But this can be prevented through several checks before you start the review. For example, ensuring that the eligibility criteria in a systematic review is well defined when designing the systematic review protocol of a systematic review article , will reduce the risk of selection bias during study selection. An effective tool called a funnel plot is designed to examine the existence of a publication bias, among studies included in a systematic review, or the tendency of authors to only publish studies with significant results, other reporting biases, and small study effects.

What Is A Funnel Plot In Systematic Reviews?

A funnel plot is a simple scatter plot of the treatment effects estimated from individual studies against a measure of the study size. The x-axis (horizontal axis) shows the results of the study, expressed as an odds or risk ratio or a percent difference, while the y-axis (vertical axis) displays the sample size or an index of precision. Other measures could also be plotted, such as reciprocals, or variances.

The scale of the y-axis is reversed; studies with higher precision are placed at the top while studies with lower precision are placed at the bottom. At the bottom where the low precision studies are placed, the points that represent the mean value of effect in each study are widely spread. The spread of these points begins to reduce, as you move upwards in the y-axis. This effect creates a plot that resembles a pyramid or an inverted funnel.

What Is The Purpose Of Funnel Plots?

A funnel plot is designed to check for the existence of publication bias, other reporting biases, and systematic heterogeneity in a systematic review. These are biases caused by the absence of information from unpublished sources (missing studies), or selective outcome reporting of a study’s result (missing outcomes). For example, the study authors may omit information that they may feel does not agree with their findings. In the absence of publication bias, the funnel plot, as its name suggests, should create a symmetrical funnel-shaped distribution. Deviations from this, like an asymmetrical plot, may indicate that there is a bias.

That said, it’s important to note that publication bias is only one of the issues examined by funnel plots. These plots can also assess small study effects, or the tendency for smaller studies to show larger treatment effects.

How to Interpret a Funnel Plot?

Funnel plots, typically, are symmetrical or asymmetrical. Here’s how to interpret them:

Symmetrical Funnel Plot

A “well-behaved” data set, one where the precision of the estimated intervention effect increases as the size of the study increases will yield a symmetric inverted funnel shape. This signifies the unlikeliness of bias.

Asymmetrical Funnel Plot

An asymmetrical funnel plot can indicate the presence of a bias, suggesting a relationship between treatment effect estimate and study precision. With these deviant shapes, you can assume the possibility of publication bias, small-study effects, or study heterogeneity. Asymmetry can also be caused by the use of an inappropriate effect measure. Whatever the cause, an asymmetric funnel plot leads to doubts about the systematic review. In this case, an investigation must be done to get to the bottom of the possible cause and correct the mistake.

Systematic reviews are one of the most rigorous research processes. But they’re not without risks as they can be prone to biases. Researchers can use tools to prevent these challenges—a funnel plot is one way to do it as it’s designed to check for publication bias, other reporting biases, and small study effects. Another great way to ensure that your systematic review is accurate, objective, and comprehensive is to use a literature review software like DistillerSR, which helps you automate each stage of your review to securely produce evidence-based research faster, and more precisely. This gives you the time and energy to focus on evaluating other information to ensure that your protocol is free from any biases.

3 Reasons to Connect

• Research Guides

Writing Literature Reviews

• Literature Review Overview
• Organizing Your Lit Review
• Tips for Writing Your Lit Review

Need Assistance?

Find your librarian, schedule a research appointment, today's hours : , what is a literature review.

A literature review ought to be a clear, concise synthesis of relevant information. A literature review should introduce the study it precedes and show how that study fits into topically related studies that already exist. Structurally, a literature review ought to be something like a funnel: start by addressing the topic broadly and gradually narrow as the review progresses.

from Literature Reviews by CU Writing Center

Why review the literature?

Reference to prior literature is a defining feature of academic and research writing. Why review the literature?

• To help you understand a research topic
• To establish the importance of a topic
• To help develop your own ideas
• To make sure you are not simply replicating research that others have already successfully completed
• To demonstrate knowledge and show how your current work is situated within, builds on, or departs from earlier publications

from Literature Review Basics from University of La Verne

Tips & Tricks

Before writing your own literature review, take a look at these resources which share helpful tips and tricks:

Lectures & Slides

• Literature Reviews | CU Writing Center
• Writing a Literature Review | CU Writing Center
• Revising a Literature Review | CU Writing Center
• Literature Reviews: How to Find and Do Them
• Literature Reviews: An Overview

How-To Guides

• Literature Reviews | Purdue OWL
• Literature Reviews | University of North Carolina
• Learn How to Write a Review of Literature | University of Wisconsin
• Literature Review: The What, Why and How-to Guide | University of Connecticut
• Literature Reviews | Florida A & M
• Conduct a Literature Review | SUNY
• Literature Review Basics | University of LaVerne

Sample Literature Reviews

• Sample Literature Reviews | University of West Florida
• Sample APA Papers: Literature Review | Purdue OWL
• Next: Organizing Your Lit Review >>
• Last Updated: Apr 24, 2020 3:12 PM
• URL: https://libguides.cedarville.edu/c.php?g=969394

Writers' Center

Eastern Washington University

Writing the Literature Review

The literature review, questions your literature review might address, tips for writing a literature review, tips for organizing your sources.

[ Back to resource home ]

[email protected] 509.359.2779

Cheney Campus   JFK Library Learning Commons

Stay Connected!

inside.ewu.edu/writerscenter Instagram  Facebook

• Occupational Therapy Sample
• Social Work Sample
• POOR Literature Review Sample This is how NOT to write a literature review.

What is a literature review?

When we hear the word "literature," we often think of great classic novels or poetry, but in this case "literature" refers to the body of work you consulted in order to make a conclusive recommendation about an issue. In other words, a literature review is a synthesis (more on synthesis below) of many articles and other published materials on a certain research topic. Depending on the field, the literature review might be a stand-alone piece or part of a larger research article.

Why write one?

By writing a literature review, you are entering into an academic conversation about an issue. You need to show that you understand what research has already been done in your field and how your own research fits into it.

1.    What do we already know in the immediate area concerned?

2.   What are the characteristics/traits of the key concepts or the main factors or variables?

3.   What are the relationships between these key concepts, factors or variables?

4.   What are the existing theories?

5.   Where are the inconsistencies or other shortcomings in our knowledge and understanding?

6.   What views need to be (further) tested?

7.   What evidence is lacking, inconclusive, contradictory or too limited?

8.   What research designs or methods seem unsatisfactory?

9.  How does this research provide context for my own work? 

Synthesis involves but goes beyond summary. Rather than simply summarizing sources (as an annotated bibliography does), synthesis also does the following:

• shows the relationships between sources (how are they similar or different, how do they build off one another, where are the gaps)
• shows how sources relate to your own work (provides the context for your research)
• sometimes evaluates the methods or conclusions of sources

Synthesis literally means to bring together, to combine separate elements to make a cohesive whole. Here are some analogies to help with the concept of synthesis:

Credit: https://www.flickr.com/photos/pcapemax2007/8711542470

Architecture: Each main idea is an element of architecture: a shape, a material, a color. All of the elements come together to form a coherent look (synthesis).

Remember your purpose.

Keeping your own research question or goals in mind as you read will help you decide which sources to include in your review and which ones to briefly mention or leave out entirely.

Know and organize your sources.

Use a free citation manager to keep track of and categorize articles in library databases. Use different colors to highlight different "threads" or "notes" (main ideas) of your sources. Take lots of notes.

Look for threads patiently.

Be patient with yourself during this process. It will take time to read, reread, annotate, and start to see connections beween sources.

Use transition phrases that indicate synthesis.

Language is your key to showing connections between ideas on paper. Consider using transition phrases like the following, or borrow phrasing that you like from other articles. (It's not plagiarism to use common phrases.)

Organize your review like a funnel.

Start by addressing the larger context of the issue at hand. Gradually work to the more specific aspects you will be looking at. Finally, narrow into your own project and research.

Read plenty of sample literature reviews in your field.

Literature reviews vary in purpose and format from field to field, so use published literature review articles in your discipline as models. You can find a few annotated samples on this page.

• Last Updated: Apr 25, 2024 2:50 PM
• URL: https://research.ewu.edu/writers_c_lit_review

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Recommendations for...

Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials

• Related content
• Peer review
• Jonathan A C Sterne , professor 1 ,
• Alex J Sutton , professor 2 ,
• John P A Ioannidis , professor and director 3 ,
• Norma Terrin , associate professor 4 ,
• David R Jones , professor 2 ,
• Joseph Lau , professor 4 ,
• James Carpenter , reader 5 ,
• Gerta Rücker , research assistant 6 ,
• Roger M Harbord , research associate 1 ,
• Christopher H Schmid , professor 4 ,
• Jennifer Tetzlaff , research coordinator 7 ,
• Jonathan J Deeks , professor 8 ,
• Jaime Peters , research fellow 9 ,
• Petra Macaskill , associate professor 10 ,
• Guido Schwarzer , research assistant 6 ,
• Sue Duval , assistant professor 11 ,
• Douglas G Altman , professor 12 ,
• David Moher , senior scientist 7 ,
• Julian P T Higgins , senior statistician 13
• 1 School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK
• 2 Department of Health Sciences, University of Leicester, Leicester, UK
• 3 Stanford Prevention Research Center, Stanford University School of Medicine, Stanford, CA, USA
• 4 Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA, USA
• 5 Medical Statistics Unit, London School of Hygiene and Tropical Medicine, London, UK
• 6 Institute of Medical Biometry and Medical Informatics, University Medical Center Freiburg, Germany
• 7 Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
• 8 School of Health and Population Sciences, University of Birmingham, Birmingham, UK
• 9 Peninsula Medical School, University of Exeter, Exeter, UK
• 10 School of Public Health, University of Sydney, NSW, Australia
• 11 University of Minnesota School of Public Health, Minneapolis, MN, USA
• 12 Centre for Statistics in Medicine, University of Oxford, Oxford, UK
• 13 MRC Biostatistics Unit, Cambridge, UK
• Correspondence to: J A C Sterne jonathan.sterne{at}bristol.ac.uk
• Accepted 21 February 2011

Funnel plots, and tests for funnel plot asymmetry, have been widely used to examine bias in the results of meta-analyses. Funnel plot asymmetry should not be equated with publication bias, because it has a number of other possible causes. This article describes how to interpret funnel plot asymmetry, recommends appropriate tests, and explains the implications for choice of meta-analysis model

The 1997 paper describing the test for funnel plot asymmetry proposed by Egger et al 1 is one of the most cited articles in the history of BMJ . 1 Despite the recommendations contained in this and subsequent papers, 2 3 funnel plot asymmetry is often, wrongly, equated with publication or other reporting biases. The use and appropriate interpretation of funnel plots and tests for funnel plot asymmetry have been controversial because of questions about statistical validity, 4 disputes over appropriate interpretation, 3 5 6 and low power of the tests. 2

This article recommends how to examine and interpret funnel plot asymmetry (also known as small study effects 2 ) in meta-analyses of randomised controlled trials. The recommendations are based on a detailed MEDLINE review of literature published up to 2007 and discussions among methodologists, who extended and adapted guidance previously summarised in the Cochrane Handbook for Systematic Reviews of Interventions. 7

What is a funnel plot?

A funnel plot is a scatter plot of the effect estimates from individual studies against some measure of each study’s size or precision. The standard error of the effect estimate is often chosen as the measure of study size and plotted on the vertical axis 8 with a reversed scale that places the larger, most powerful studies towards the top. The effect estimates from smaller studies should scatter more widely at the bottom, with the spread narrowing among larger studies. 9 In the absence of bias and between study heterogeneity, the scatter will be due to sampling variation alone and the plot will resemble a symmetrical inverted funnel (fig 1 ⇓ ). A triangle centred on a fixed effect summary estimate and extending 1.96 standard errors either side will include about 95% of studies if no bias is present and the fixed effect assumption (that the true treatment effect is the same in each study) is valid. The appendix on bmj.com discusses choice of axis in funnel plots.

Fig 1 Example of symmetrical funnel plot. The outer dashed lines indicate the triangular region within which 95% of studies are expected to lie in the absence of both biases and heterogeneity (fixed effect summary log odds ratio±1.96×standard error of summary log odds ratio). The solid vertical line corresponds to no intervention effect

• Download figure
• Open in new tab
• Download powerpoint

Implications of heterogeneity, reporting bias, and chance

Heterogeneity, reporting bias, and chance may all lead to asymmetry or other shapes in funnel plots (box). Funnel plot asymmetry may also be an artefact of the choice of statistics being plotted (see appendix). The presence of any shape in a funnel plot is contingent on the studies having a range of standard errors, since otherwise they would lie on a horizontal line.

Box 1: Possible sources of asymmetry in funnel plots (adapted from Egger et al 1 )

Reporting biases.

Publication bias:

 Delayed publication (also known as time lag or pipeline) bias

 Location biases (eg, language bias, citation bias, multiple publication bias)

Selective outcome reporting

Selective analysis reporting

Poor methodological quality leading to spuriously inflated effects in smaller studies

Poor methodological design

Inadequate analysis

True heterogeneity

Size of effect differs according to study size (eg, because of differences in the intensity of interventions or in underlying risk between studies of different sizes)

Artefactual

In some circumstances, sampling variation can lead to an association between the intervention effect and its standard error

Asymmetry may occur by chance, which motivates the use of asymmetry tests

Heterogeneity

Statistical heterogeneity refers to differences between study results beyond those attributable to chance. It may arise because of clinical differences between studies (for example, setting, types of participants, or implementation of the intervention) or methodological differences (such as extent of control over bias). A random effects model is often used to incorporate heterogeneity in meta-analyses. If the heterogeneity fits with the assumptions of this model, a funnel plot will be symmetrical but with additional horizontal scatter. If heterogeneity is large it may overwhelm the sampling error, so that the plot appears cylindrical.

Heterogeneity will lead to funnel plot asymmetry if it induces a correlation between study sizes and intervention effects. 5 For example, substantial benefit may be seen only in high risk patients, and these may be preferentially included in early, small studies. 10 Or the intervention may have been implemented less thoroughly in larger studies, resulting in smaller effect estimates compared with smaller studies. 11

Figure 2 ⇓ shows funnel plot asymmetry arising from heterogeneity that is due entirely to there being three distinct subgroups of studies, each with a different intervention effect. 12 The separate funnels for each subgroup are symmetrical. Unfortunately, in practice, important sources of heterogeneity are often unknown.

Fig 2 Illustration of funnel plot asymmetry due to heterogeneity, in the form of three distinct subgroups of studies. Funnel plot including all studies (top left) shows clear asymmetry (P<0.001 from Egger test for funnel plot asymmetry). P values for each subgroup are all >0.49.

Differences in methodological quality may also cause heterogeneity and lead to funnel plot asymmetry. Smaller studies tend to be conducted and analysed with less methodological rigour than larger studies, 13 and trials of lower quality also tend to show larger intervention effects. 14 15

Reporting bias

Reporting biases arise when the dissemination of research findings is influenced by the nature and direction of results. Statistically significant “positive” results are more likely to be published, published rapidly, published in English, published more than once, published in high impact journals, and cited by others. 16 17 18 19 Data that would lead to negative results may be filtered, manipulated, or presented in such a way that they become positive. 14 20

Reporting biases can have three types of consequence for a meta-analysis:

A systematic review may fail to locate an eligible study because all information about it is suppressed or hard to find (publication bias)

A located study may not provide usable data for the outcome of interest because the study authors did not consider the result sufficiently interesting (selective outcome reporting)

A located study may provide biased results for some outcome—for example, by presenting the result with the smallest P value or largest effect estimate after trying several analysis methods (selective analysis reporting).

These biases may cause funnel plot asymmetry if statistically significant results suggesting a beneficial effect are more likely to be published than non-significant results. Such asymmetry may be exaggerated if there is a further tendency for smaller studies to be more prone to selective suppression of results than larger studies. This is often assumed to be the case for randomised trials. For instance, it is probably more difficult to make a large study disappear without trace, while a small study can easily be lost in a file drawer. 21 The same may apply to specific outcomes—for example, it is difficult not to report on mortality or myocardial infarction if these are outcomes of a large study.

Smaller studies have more sampling error in their effect estimates. Thus even though the risk of a false positive significant finding is the same, multiple analyses are more likely to yield a large effect estimate that may seem worth publishing. However, biases may not act this way in real life; funnel plots could be symmetrical even in the presence of publication bias or selective outcome reporting 19 22 —for example, if the published findings point to effects in different directions but unreported results indicate neither direction. Alternatively, bias may have affected few studies and therefore not cause glaring asymmetry.

The role of chance is critical for interpretation of funnel plots because most meta-analyses of randomised trials in healthcare contain few studies. 2 Investigations of relations across studies in a meta-analysis are seriously prone to false positive findings when there is a small number of studies and heterogeneity across studies, 23 and this may affect funnel plot symmetry.

Interpreting funnel plot asymmetry

Authors of systematic reviews should distinguish between possible reasons for funnel plot asymmetry (box 1). Knowledge of the intervention, and the circumstances in which it was implemented in different studies, can help identify causes of asymmetry in funnel plots, which should also be interpreted in the context of susceptibility to biases of research in the field of interest. Potential conflicts of interest, whether outcomes and analyses have been standardised, and extent of trial registration may need to be considered. For example, studies of antidepressants generate substantial conflicts of interest because the drugs generate vast sales revenues. Furthermore, there are hundreds of outcome scales, analyses can be very flexible, and trial registration was uncommon until recently. 24 Conversely, in a prospective meta-analysis where all data are included and all analyses fully standardised and conducted according to a predetermined protocol, publication or reporting biases cannot exist. Reporting bias is therefore more likely to be a cause of an asymmetric plot in the first situation than in the second.

Terrin et al found that researchers were poor at identifying publication bias from funnel plots. 5 Including contour lines corresponding to perceived milestones of statistical significance (P=0.01, 0.05, 0.1, etc) may aid visual interpretation. 25 If studies seem to be missing in areas of non-significance (fig 3 ⇓ , top) then asymmetry may be due to reporting bias, although other explanations should still be considered. If the supposed missing studies are in areas of higher significance or in a direction likely to be considered desirable to their authors (fig 3 ⇓ , bottom), asymmetry is probably due to factors other than reporting bias.

Fig 3 Contour enhanced funnel plots. In the top diagram there is a suggestion of missing studies in the middle and right of the plot, broadly in the white area of non-significance, making publication bias plausible. In the bottom diagram there is a suggestion of missing studies on the bottom left hand side of the plot. Since most of this area contains regions of high significance, publication bias is unlikely to be the underlying cause of asymmetry

Statistical tests for funnel plot asymmetry

A test for funnel plot asymmetry (sometimes referred to as a test for small study effects) examines whether the association between estimated intervention effects and a measure of study size is greater than might be expected to occur by chance. These tests typically have low power, so even when a test does not provide evidence of asymmetry, bias cannot be excluded. For outcomes measured on a continuous scale a test based on a weighted linear regression of the effect estimates on their standard errors is straightforward. 1 When outcomes are dichotomous and intervention effects are expressed as odds ratios, this corresponds to an inverse variance weighted linear regression of the log odds ratio on its standard error. 2 Unfortunately, there are statistical problems because the standard error of the log odds ratio is mathematically linked to the size of the odds ratio, even in the absence of small study effects. 2 4 Many authors have therefore proposed alternative tests (see appendix on bmj.com). 4 26 27 28

Because it is impossible to know the precise mechanism(s) leading to funnel plot asymmetry, simulation studies (in which tests are evaluated on large numbers of computer generated datasets) are required to evaluate test characteristics. Most have examined a range of assumptions about the extent of reporting bias by selectively removing studies from simulated datasets. 26 27 28 After reviewing the results of these studies, and based on theoretical considerations, we formulated recommendations on testing for funnel plot asymmetry (box 2). The appendix describes the proposed tests, explains the reasons that some were not recommended, and discusses funnel plots for intervention effects measured as risk ratios, risk differences, and standardised mean differences. Our recommendations imply that tests for funnel plot asymmetry should be used in only a minority of meta-analyses. 29

Box 2: Recommendations on testing for funnel plot asymmetry

All types of outcome.

As a rule of thumb, tests for funnel plot asymmetry should not be used when there are fewer than 10 studies in the meta-analysis because test power is usually too low to distinguish chance from real asymmetry. (The lower the power of a test, the higher the proportion of “statistically significant” results in which there is in reality no association between study size and intervention effects). In some situations—for example, when there is substantial heterogeneity—the minimum number of studies may be substantially more than 10

Test results should be interpreted in the context of visual inspection of funnel plots— for example, are there studies with markedly different intervention effect estimates or studies that are highly influential in the asymmetry test? Even if an asymmetry test is statistically significant, publication bias can probably be excluded if small studies tend to lead to lower estimates of benefit than larger studies or if there are no studies with significant results

When there is evidence of funnel plot asymmetry, publication bias is only one possible explanation (see box 1)

As far as possible, testing strategy should be specified in advance: choice of test may depend on the degree of heterogeneity observed. Applying and reporting many tests is discouraged: if more than one test is used, all test results should be reported

Tests for funnel plot asymmetry should not be used if the standard errors of the intervention effect estimates are all similar (the studies are of similar sizes)

Continuous outcomes with intervention effects measured as mean differences

The test proposed by Egger et al may be used to test for funnel plot asymmetry. 1 There is no reason to prefer more recently proposed tests, although their relative advantages and disadvantages have not been formally examined. General considerations suggest that the power will be greater than for dichotomous outcomes but that use of the test with substantially fewer than 10 studies would be unwise

Dichotomous outcomes with intervention effects measured as odds ratios

The tests proposed by Harbord et al 26 and Peters et al 27 avoid the mathematical association between the log odds ratio and its standard error when there is a substantial intervention effect while retaining power compared with alternative tests. However, false positive results may still occur if there is substantial between study heterogeneity

If there is substantial between study heterogeneity (the estimated heterogeneity variance of log odds ratios, τ 2 , is >0.1) only the arcsine test including random effects, proposed by Rücker et al, has been shown to work reasonably well. 28 However, it is slightly conservative in the absence of heterogeneity and its interpretation is less familiar than for other tests because it is based on an arcsine transformation.

When τ 2 is <0.1, one of the tests proposed by Harbord et al, 26 Peters et al, 27 or Rücker et al 28 can be used. Test performance generally deteriorates as τ 2 increases.

Funnel plots and meta-analysis models

Fixed and random effects models.

Funnel plots can help guide choice of meta-analysis method. Random effects meta-analyses weight studies relatively more equally than fixed effect analyses by incorporating the between study variance into the denominator of each weight. If effect estimates are related to standard errors (funnel plot asymmetry), the random effects estimate will be pulled more towards findings from smaller studies than the fixed effect estimate will be. Random effects models can thus have undesirable consequences and are not always conservative. 30

The trials of intravenous magnesium after myocardial infarction provide an extreme example of the differences between fixed and random effects analyses that can arise in the presence of funnel plot asymmetry. 31 Beneficial effects on mortality, found in a meta-analysis of small studies, 32 were subsequently contradicted when the very large ISIS-4 study found no evidence of benefit. 33 A contour enhanced funnel plot (fig 4 ⇓ ) gives a clear visual impression of asymmetry, which is confirmed by small P values from the Harbord and Peters tests (P<0.001 and P=0.002 respectively).

Fig 4 Contour enhanced funnel plot for trials of the effect of intravenous magnesium on mortality after myocardial infarction

Figure 5 ⇓ shows that in a fixed effect analysis ISIS-4 receives 90% of the weight, and there is no evidence of a beneficial effect. However, there is clear evidence of between study heterogeneity (P<0.001, I 2 =68%), and in a random effects analysis the small studies dominate so that intervention appears beneficial. To interpret the accumulated evidence, it is necessary to make a judgment about the validity or relevance of the combined evidence from the smaller studies compared with that from ISIS-4. The contour enhanced funnel plot suggests that publication bias does not completely explain the asymmetry, since many of the beneficial effects reported from smaller studies were not significant. Plausible explanations for these results are that methodological flaws in the smaller studies, or changes in the standard of care (widespread adoption of treatments such as aspirin, heparin, and thrombolysis), led to apparent beneficial effects of magnesium. This belief was reinforced by the subsequent publication of the MAGIC trial, in which magnesium added to these treatments which also found no evidence of benefit on mortality (odds ratio 1.0, 95% confidence interval 0.8 to 1.1). 34

Fig 5 Comparison of fixed and random effects meta-analytical estimates of the effect of intravenous magnesium on mortality after myocardial infarction

We recommend that when review authors are concerned about funnel plot asymmetry in a meta-analysis with evidence of between study heterogeneity, they should compare the fixed and random effects estimates of the intervention effect. If the random effects estimate is more beneficial, authors should consider whether it is plausible that the intervention is more effective in smaller studies. Formal investigations of heterogeneity of effects may reveal explanations for funnel plot asymmetry, in which case presentation of results should focus on these. If larger studies tend to be methodologically superior to smaller studies, or were conducted in circumstances more typical of the use of the intervention in practice, it may be appropriate to include only larger studies in the meta-analysis.

Extrapolation of a funnel plot regression line

An assumed relation between susceptibility to bias and study size can be exploited by extrapolating within a funnel plot. When funnel plot asymmetry is due to bias rather than substantive heterogeneity, it is usually assumed that results from larger studies are more believable than those from smaller studies because they are less susceptible to methodological flaws or reporting biases. Extrapolating a regression line on a funnel plot to minimum bias (maximum sample size) produces a meta-analytical estimate that can be regarded as corrected for such biases. 35 36 37 However, because it is difficult to distinguish between asymmetry due to bias and asymmetry due to heterogeneity or chance, the broad applicability of such approaches is uncertain. Further approaches to adjusting for publication bias are described and discussed in the appendix.

Reporting biases are one of a number of possible explanations for the associations between study size and effect size that are displayed in asymmetric funnel plots. Examining and testing for funnel plot asymmetry, when appropriate, is an important means of addressing bias in meta-analyses, but the multiple causes of asymmetry and limited power of asymmetry tests mean that other ways to address reporting biases are also of importance. Searches of online trial registries can identify unpublished trials, although they do not currently guarantee access to trial protocols and results. When there are no registered but unpublished trials, and the outcome of interest is reported by all trials, restricting meta-analyses to registered trials should preclude publication bias. Recent comparisons of results of published trials with those submitted for regulatory approval have also provided clear evidence of reporting bias. 38 39 Methods for dealing with selective reporting of outcomes have been described elsewhere. 40

Our recommendations apply to meta-analyses of randomised trials, and their applicability in other contexts such as meta-analyses of epidemiological or diagnostic test studies is unclear. 41 The performance of tests for funnel plot asymmetry in these contexts is likely to differ from that in meta-analyses of randomised trials. Further factors, such as confounding and precision of measurements, may cause a relation between study size and effect estimates in observational studies. For example, large studies based on routinely collected data might not fully control confounding compared with smaller, purpose designed studies that collected a wide range of potential confounding variables. Alternatively, larger studies might use self reported exposure levels, which are more error prone, while smaller studies used precise measuring instruments. However, simulation studies have usually not considered such situations. An exception is for diagnostic studies, where large imbalances in group sizes and substantial odds ratios lead to poor performance of some tests: that proposed by Deeks et al was designed for use in this context. 4

Summary points

Inferences on the presence of bias or heterogeneity should consider different causes of funnel plot asymmetry and should not be based on visual inspection of funnel plots alone

They should be informed by contextual factors, including the plausibility of publication bias as an explanation for the asymmetry

Testing for funnel plot asymmetry should follow the recommendations detailed in this article

The fixed and random effects estimates of the intervention effect should be compared when funnel plot asymmetry exists in a meta-analysis with between study heterogeneity

Cite this as: BMJ 2011;342:d4002

Contributors: All authors contributed to the drafting and editing of the manuscript. DA, JC, JD, RMH, JPTH, JPAI, DRJ, DM, JP, GR, JACS, AJS and JT contributed to the chapter in the Cochrane Handbook for Systematic Reviews of Interventions on which our recommendations on testing for funnel plot asymmetry are based. JACS will act as guarantor.

Funding: Funded in part by the Cochrane Collaboration Bias Methods Group, which receives infrastructure funding as part of a commitment by the Canadian Institutes of Health Research (CIHR) and the Canadian Agency for Drugs and Technologies in Health (CADTH) to fund Canadian based Cochrane entities. This supports dissemination activities, web hosting, travel, training, workshops and a full time coordinator position. JPTH was funded by MRC Grant U.1052.00.011. DGA is supported by Cancer Research UK. GR was supported by a grant from Deutsche Forschungsgemeinschaft (FOR 534 Schw 821/2-2).

Competing interests. JC, JJD, SD, RMH, JPAI, DRJ, PM, JP, GR, GS, JACS and AJS are all authors on papers proposing tests for funnel plot asymmetry, but have no commercial interests in the use of these tests. All authors have completed the ICJME unified disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that they have no financial or non-financial interests that may be relevant to the submitted work.

Provenance and peer review: Not commissioned; externally peer reviewed.

• ↵ Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997 ; 315 : 629 -34. OpenUrl Abstract / FREE Full Text
• ↵ Sterne JAC, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol 2000 ; 53 : 1119 -29. OpenUrl CrossRef PubMed Web of Science
• ↵ Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ 2006 ; 333 : 597 -600. OpenUrl FREE Full Text
• ↵ Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol 2005 ; 58 : 882 -93. OpenUrl CrossRef PubMed Web of Science
• ↵ Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol 2005 ; 58 : 894 -901. OpenUrl CrossRef PubMed Web of Science
• ↵ Ioannidis JP. Interpretation of tests of heterogeneity and bias in meta-analysis. J Eval Clin Pract 2008 ; 14 : 951 -7. OpenUrl CrossRef PubMed Web of Science
• ↵ Sterne JAC, Egger M, Moher D. Addressing reporting biases. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions . Wiley, 2008 .
• ↵ Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol 2001 ; 54 : 1046 -55. OpenUrl CrossRef PubMed Web of Science
• ↵ Begg CB, Berlin JA. Publication bias: a problem in interpreting medical data. J R Statist Soc A 1988 ; 151 : 419 -63. OpenUrl CrossRef
• ↵ Davey Smith G, Egger M. Who benefits from medical interventions? Treating low risk patients can be a high risk strategy. BMJ 1994 ; 308 : 72 -4. OpenUrl FREE Full Text
• ↵ Stuck AE, Siu AL, Wieland GD, Adams J, Rubenstein LZ. Comprehensive geriatric assessment: a meta-analysis of controlled trials. Lancet 1993 ; 342 : 1032 -6. OpenUrl CrossRef PubMed Web of Science
• ↵ Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L, Moreno SG. Assessing publication bias in meta-analyses in the presence of between-study heterogeneity. J R Statist Soc A 2010 ; 173 : 575 -91. OpenUrl CrossRef
• ↵ Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 2003 ; 7 : 1 -68. OpenUrl PubMed
• ↵ Ioannidis JP. Why most discovered true associations are inflated. Epidemiology 2008 ; 19 : 640 -8. OpenUrl CrossRef PubMed Web of Science
• ↵ Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008 ; 336 : 601 -5. OpenUrl Abstract / FREE Full Text
• ↵ Hopewell S, Clarke M, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database Syst Rev 2007 ; 2 : MR000011 . OpenUrl PubMed
• ↵ Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009 ; 1 : MR000006 . OpenUrl PubMed
• ↵ Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton J, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 2010 ; 14 : iii ,ix-iii,193. OpenUrl PubMed Web of Science
• ↵ Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 2008 ; 3 : e3081 . OpenUrl CrossRef PubMed
• ↵ Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008 ; 358 : 252 -60. OpenUrl CrossRef PubMed Web of Science
• ↵ Rosenthal R. The “file drawer” problem and tolerance for null results. Psychol Bull 1979 ; 86 : 638 -41. OpenUrl CrossRef Web of Science
• ↵ Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004 ; 291 : 2457 -65. OpenUrl CrossRef PubMed Web of Science
• ↵ Higgins JP, Thompson SG. Controlling the risk of spurious findings from meta-regression. Stat Med 2004 ; 23 : 1663 -82. OpenUrl CrossRef PubMed Web of Science
• ↵ Ioannidis JP. Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized trials? Philos Ethics Humanit Med 2008 ; 3 : 14 . OpenUrl CrossRef PubMed
• ↵ Peters J, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol 2008 ; 61 : 991 -6. OpenUrl CrossRef PubMed Web of Science
• ↵ Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med 2006 ; 25 : 3443 -57. OpenUrl CrossRef PubMed Web of Science
• ↵ Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA 2006 ; 295 : 676 -80. OpenUrl CrossRef PubMed Web of Science
• ↵ Rücker G, Schwarzer G, Carpenter J. Arcsine test for publication bias in meta-analyses with binary outcomes. Stat Med 2008 ; 27 : 746 -63. OpenUrl CrossRef PubMed Web of Science
• ↵ Ioannidis JP, Trikalinos TA. The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ 2007 ; 176 : 1091 -6. OpenUrl Abstract / FREE Full Text
• ↵ Poole C, Greenland S. Random-effects meta-analyses are not always conservative. Am J Epidemiol 1999 ; 150 : 469 -75. OpenUrl Abstract / FREE Full Text
• ↵ Egger M, Davey Smith G. Misleading meta-analysis. Lessons from an “effective, safe, simple” intervention that wasn’t. BMJ 1995 ; 310 : 752 -4. OpenUrl FREE Full Text
• ↵ Teo KK, Yusuf S, Collins R, Held PH, Peto R. Effects of intravenous magnesium in suspected acute myocardial infarction: overview of randomised trials. BMJ 1991 ; 303 : 1499 -503. OpenUrl Abstract / FREE Full Text
• ↵ ISIS-4 (Fourth International Study of Infarct Survival) Collaborative Group. ISIS-4: a randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction. Lancet 1995 ; 345 : 669 -85. OpenUrl CrossRef PubMed Web of Science
• ↵ Early administration of intravenous magnesium to high-risk patients with acute myocardial infarction in the Magnesium in Coronaries (MAGIC) Trial: a randomised controlled trial. Lancet 2002 ; 360 : 1189 -96. OpenUrl CrossRef PubMed Web of Science
• ↵ Shang A, Huwiler-Muntener K, Nartey L, Jüni P, Dörig S, Stene JA, et al. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Lancet 2005 ; 366 : 726 -32. OpenUrl CrossRef PubMed Web of Science
• ↵ Moreno SG, Sutton AJ, Ades AE, Stanley TD, Abrams KR, Peters JL, et al. Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Med Res Methodol 2009 ; 9 : 2 . OpenUrl CrossRef PubMed
• ↵ Rucker G, Schwarzer G, Carpenter JR, Binder H, Schumacher M. Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis. Biostatistics 2011 ; 12 : 122 -42. OpenUrl Abstract / FREE Full Text
• ↵ Moreno SG, Sutton AJ, Turner EH, Abrams KR, Cooper NJ, Palmer TM, et al. Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. BMJ 2009 ; 339 : b2981 . OpenUrl Abstract / FREE Full Text
• ↵ Eyding D, Lelgemann M, Grouven U, Härter M, Kromp M, Kaiser T, et al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 2010 ; 341 : c4737 . OpenUrl Abstract / FREE Full Text
• ↵ Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010 ; 340 : c365 . OpenUrl Abstract / FREE Full Text
• ↵ Egger M, Schneider M, Davey Smith G. Spurious precision? Meta-analysis of observational studies. BMJ 1998 ; 316 : 140 -4. OpenUrl FREE Full Text

• About LiveInnovation.org
• Prof. Dr. Francisco Tigre Moura
• Academic Research
• Live AM: Artist Monitor
• Live FM: Fan Monitor
• Books and e-Books
• Media/Events
• Consumer Behavior
• Marketing Research
• Statistics Support
• Thesis Writing

• Research Support

How To Write an Excellent LITERATURE REVIEW For Your Research

Let me guess:  You know what you will write about, you have mastered Chapter 1 (Introduction) but you don’t know how to move forward with your Chapter 2 (literature review)?

Alright, alright… I feel your pain. But that’s why I’ve written down some recommendations that hopefully will be useful for you!

INITIAL WARNING

Look, I need to be honest with you here. Writing your theoretical background will not be the most entertaining thing you have done in your life.

Writing a theoretical background is definitely not like sitting under the sun on a tropical beach with the sea water washing your feet exactly when the waiter comes to serve you with a dish of fresh fried fish and a bucket filled with extremely cold beer while your gorgeous smiley girlfriend (wearing a bikini) is singing beautifully “God Only Knows” (Best Beach Boys song). If you don’t know the band or the song, shame on you!) and this is only the first day of vacation.

Writing a literature review is more like going to your City Hall at 7.30am on a hot summer Monday and standing in line for 2 hours at an old building with no AC to pay your taxes to an impolite government employee.

Is it fun? NO . Do you have any choice? NO . And is it important? VERY .

Summing up: Head down, take a deep breath and get it done with!

So how do you do it? SIMPLE. I suggest you do the following:

1. FIRST OF ALL: In this chapter you will discuss only LITERATURE (theoretical concepts) related to your topic.

What does that mean? Assume you want to understand the buying criteria of consumers in regards to Smartphones. Assume this is the title of your Thesis: “ Understanding Consumers’ Evaluation of Cellphone Attributes during Their Purchase Decision Making Process ”.

In this chapter you will NOT discuss cellphones in specific! (e.g. Describe types, brands, etc). This is simply your CONTEXT . In this chapter you will discuss THEORY ( applied to your context! ).

FOR EXAMPLE: In this case, you could discuss the following topics:

• Decision Making Process (BROAD TOPIC)
• Product Value Perception
• Consumer Motivation and Involvement

3.1 Consumer Involvement with Electronic Products

• Attribute Evaluation

4.1 Fischbein’s Theory: Multi-attribute Model

• Smartphone Attribute Evaluation (SPECIFIC TOPIC)
• Research Gap

6.1 Research Aim/Question

With this structure did we discuss types of cellphones or their brands? NO ! Again, NO ! There are only THEORETICAL concepts listed and related to the context of the study!

Also, did you notice that the list goes from the BROADEST (most general) topic to the MOST SPECIFIC topic ?

Exactly! That’s the second recommendation!

Video Support: Literature Review

In case you are enjoying the article, do not forget to watch the video with further support on how to write the literature review in your thesis.

 2. SECOND OF ALL: Structure the SCOPE of your study.

Think of your theoretical background (or Literature review) as a FUNNEL . Yup, you heard it. A FUNNEL!   You must structure it so that you can cover it from the broadest topic until the most specific !

Now here it is difficult for me to help because this is very topic specific. So use your imagination and please try to envision the figure below as if it was a FUNNEL.

 (I know it has nothing to do with a funnel but that’s the maximum I could manage, so don’t judge my funnel development skills!).

But HOW do you know how to structure your concepts from the broadest to the most specific? This depends on EACH project. So, sorry but I can’t help you with this.

Discuss it with your SUPERVISOR! I’m just a friend trying to help!

3. THIRD OF ALL: Conclude the chapter by presenting your RESEARCH GAP and RESEARCH QUESTION (or Aim).

The theoretical background should BROADLY enable the reader to do the following:

• Become familiar with the main topics related to your study (So that he/she can understand what you have developed!)
• Understand the logic of how you got to your research question or research aim (depends how you framed it!).

Thus, I recommend you to CONCLUDE your theoretical background by discussing your research GAP and your RESEARCH question.

• RESEARCH GAP

In this section you will generally summarize the topics discussed before and highlight what needs to be researched in your field ( the RESEARCH GAP you have identified! )

• RESEARCH AIM/QUESTION

In this sub-section you will remind the reader what your research aim is. Or you may have framed the aim into a research question and will present it here.

Basically , the reader has to read your RESEARCH GAP section and think: “ OHHH… Now I get it how all concepts CONNECT to each other and what you are trying to investigate ”!

4. FOURTH OF ALL: Use RELIABLE sources and WISELY!

A) Use almost ONLY scientific articles to describe concepts and previous studies! And why

• Scientific articles are peer reviewed, thus credible sources! (Generally speaking, I know).
• Check HERE to see their ranking and use HIGH ranked journals!

B) DO NOT use commercial websites to describe concepts!

• BBC, CNN and others are not scientific sources! But what if they mention a study? Then remember: ALWAYS LOOK FOR THE PRIMARY SOURCE OF SECONDARY DATA!

C) Avoid OVERUSING the same author to explain a concept.

Have you used the same source three times or more on the same page or section?

• If so, basically you are telling your reader: I am too LAZY to look for other sources so I’m just going to cite this same authors as much as possible. And of course, that is NOT a good sign!

In general you must have done the following in this chapter:

• In this chapter you have discussed only LITERATURE (theoretical concepts) related to your topic.
• You structured the SCOPE of your study.
• You concluded the chapter by presenting your RESEARCH GAP and RESEARCH QUESTION (or Aim).
• You used only RELIABLE sources and WISELY!

  So now that you are DONE with Chapter Two, you can focus on the methodology of the thesis!

(In case you want to thank me later: I like cappuccino and Formula 1).

DOWNLOAD THE RECOMMENDATIONS

Did you like this article? Would like to have these recommendations with you while studying?

GREAT! Simply download the file here with all details: LiveInnovation.org - Developing a Theoretical Background Literature Review of a Thesis.pdf

In case you would like to have more research suggestions, check our research resources section .

RELATED ARTICLES MORE FROM AUTHOR

SOUNDS LIKE A THESIS is now available on Spotify!

Download Our e-Book: “Sounds Like A Thesis”

SPSS Tutorial Series on YouTube: Learn Quickly and Easily

Privacy overview.

This is an necessary category.

This is an non-necessary category.

Adopting a Funnel Strategy and Using Mind Mapping to Visualize the Research Design

• First Online: 25 May 2023

Cite this chapter

• Uche M. Mbanaso 4 ,
• Lucienne Abrahams 5 &
• Kennedy Chinedu Okafor 6  

481 Accesses

This chapter explains the need to adopt a research strategy and illustrates the funnel strategy that enables the researcher to move from identifying the flaws, or defects or gaps that require attention to scoping the problem statement, finalizing the research proposal as a foundation for data collection and analysis and conducting the experiments or fieldwork, through to the point of analysis and thesis writing. It presents the funnel strategy as a roadmap that can be applied to completing the various phases of the research. The chapter demonstrates the application of mind mapping techniques (graphical representations of what to do and how to do it) to the identified research problem. It sets out a series of major activities as part of the mind mapping process. It presents a series of visuals, for example, artificial intelligence and robotics sub-topics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as EPUB and PDF
• Read on any device
• Instant download
• Own it forever
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

If you find that any of the following material is not downloadable at the listed url, please search on ResearchGate at https://www.researchgate.net/ or, alternatively, search on the full name of the article, book, or paper.

Bibliography

APA. (n.d.). APA style guide . https://apastyle.apa.org/

Carlsson, S. A. (2006). Design science research in information systems: A critical realist perspective. In ACIS 2006 Proceedings, Australia . https://aisel.aisnet.org/acis2006/40/

Crowe, M., & Sheppard, L. (2016). Mind mapping research methods. Quality & Quantity, 46 , 1493–1504. https://doi.org/10.1007/s11135-011-9463-8

Article   Google Scholar  

Ellis, T., & Levy, Y. (2009). Towards a guide for novice researchers on research methodology: Review and proposed methods. Issues in Informing Science and Information Technology, 6 , 323–337. https://doi.org/10.28945/1062

Ernest, P. (1996). Varieties of constructivism: A framework for comparison. In L. Steffe, P. Nesher, P. Cobb, G. Goldin, & B. Greer (Eds.), Theories of mathematical learning . Routledge. https://doi.org/10.4324/9780203053126

Chapter   Google Scholar  

Hassani, H. (2017). Research methods in computer science: The challenges and issues . https://arxiv.org/pdf/1703.04080.pdf

Hauberg, J. (2011). Research by design: A research strategy. Architecture & Education Journal, 5 . https://www.researchgate.net/publication/279466514_Research_by_design_a_research_strategy

Hevner, A., March, S., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28 (1), 75–105. https://www.researchgate.net/publication/201168946_Design_Science_in_Information_Systems_Research

Indulska, M., & Recker, J. (2010). Design science in IS research: A literature analysis. In S. Gregor & D. Hart (Eds.), Information systems foundations: The role of design science . ANU Press. https://doi.org/10.22459/isf.12.2010.13

MacIntyre, S., Dalkir, K., Paul, P., & Kitimbo, I. (2014). Utilizing evidence-based lessons learned for enhanced organizational innovation and change . Business Science Reference.

Google Scholar  

McLaren, T., & Buijs, P. (2011). A design science approach for developing information systems research instruments. In H. Jain, A. Sinha & P. Vitharana (Eds.), Lecture notes in computer science: Vol. 6629. Service-oriented perspectives in design science research (pp. 1–10). Springer. http://www.rug.nl/staff/p.buijs/design_science_approach_for_developing_isr_instruments.pdf

Mosina, C. (2020). Understanding the diffusion of the Internet: Redesigning the global diffusion of the Internet framework . [Research report, Master of Arts in ICT Policy and Regulation]. LINK Centre, University of the Witwatersrand. http://wiredspace.wits.ac.za/handle/10539/30723

Oates, B. J. (2006). Researching information systems and computing . SAGE Publications. https://books.google.co.za/books/about/Researching_Information_Systems_and_Comp.html?id=ztrj8aph-4sC

Pajares, F. (2007). The elements of a proposal . Emory University. https://www.uky.edu/~eushe2/Pajares/proposal.html

Riggins, F. J., & Wamba, S. F. (2015). Research directions on the adoption, usage, and impact of the internet of things through the use of big data analytics. Proceedings of the 2015 48th Hawaii International Conference on Systems Science, USA , 1531–1540. https://doi.org/10.1109/HICSS.2015.186

Salhin, A., Kyiu, A., Taheri, B., Porter, C., Valantasis-Kanellos, N., & König, C. (2016). Quantitative data gathering methods and techniques. In A. Paterson, D. Leung, W. Jackson, R. MacIntosh, & K. D. O’Gorman (Eds.), Research methods for accounting and finance . Goodfellow Publishers. https://doi.org/10.23912/978-1-910158-88-3-3226

Software Testing Help. (2021, March 27). 10 best mind mapping software in 2021 . [website]. https://www.softwaretestinghelp.com/mind-map-software/

ThinkBuzan Ltd. (2010). Mind mapping: Scientific research and studies . [online]. https://www.slideshare.net/elsavonlicy/mind-mapping-evidence-report

Wilson, K., Copeland-Solas, E., & Guthrie-Dixon, N. (2016). A preliminary study on the use of mind mapping as a visual-learning strategy, in general education science classes for Arabic speakers in The United Arab Emirates. Journal of the Scholarship of Teaching and Learning, 16 (1), 31–52. https://doi.org/10.14434/josotl.v16i1.19181

Wolcott, P., Press, L., McHenry, W., Goodman, S., & Foster, W. (2001). A framework for assessing the global diffusion of the Internet. Journal of the Association for Information Systems, 2 . https://doi.org/10.17705/1jais.00018

Download references

Author information

Authors and affiliations.

Centre for Cybersecurity Studies, Nasarawa State University, Keffi, Nigeria

Uche M. Mbanaso

LINK Centre, University of the Witwatersrand, Johannesburg, South Africa

Lucienne Abrahams

Department of Mechatronics Engineering, Federal University of Technology, Owerri, Nigeria

Kennedy Chinedu Okafor

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Mbanaso, U.M., Abrahams, L., Okafor, K.C. (2023). Adopting a Funnel Strategy and Using Mind Mapping to Visualize the Research Design. In: Research Techniques for Computer Science, Information Systems and Cybersecurity. Springer, Cham. https://doi.org/10.1007/978-3-031-30031-8_4

Download citation

DOI : https://doi.org/10.1007/978-3-031-30031-8_4

Published : 25 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-30030-1

Online ISBN : 978-3-031-30031-8

eBook Packages : Engineering Engineering (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Search This Site All UCSD Sites Faculty/Staff Search Term
• Contact & Directions
• Climate Statement
• Cognitive Behavioral Neuroscience
• Cognitive Psychology
• Developmental Psychology
• Social Psychology
• Adjunct Faculty
• Non-Senate Instructors
• Researchers
• Psychology Grads
• Affiliated Grads
• New and Prospective Students
• Honors Program
• Experiential Learning
• Programs & Events
• Psi Chi / Psychology Club
• Prospective PhD Students
• Current PhD Students
• Area Brown Bags
• Colloquium Series
• Anderson Distinguished Lecture Series
• Speaker Videos
• Undergraduate Program
• Academic and Writing Resources

Writing Research Papers

• Writing a Literature Review

When writing a research paper on a specific topic, you will often need to include an overview of any prior research that has been conducted on that topic.  For example, if your research paper is describing an experiment on fear conditioning, then you will probably need to provide an overview of prior research on fear conditioning.  That overview is typically known as a literature review.  

Please note that a full-length literature review article may be suitable for fulfilling the requirements for the Psychology B.S. Degree Research Paper .  For further details, please check with your faculty advisor.

Different Types of Literature Reviews

Literature reviews come in many forms.  They can be part of a research paper, for example as part of the Introduction section.  They can be one chapter of a doctoral dissertation.  Literature reviews can also “stand alone” as separate articles by themselves.  For instance, some journals such as Annual Review of Psychology , Psychological Bulletin , and others typically publish full-length review articles.  Similarly, in courses at UCSD, you may be asked to write a research paper that is itself a literature review (such as, with an instructor’s permission, in fulfillment of the B.S. Degree Research Paper requirement). Alternatively, you may be expected to include a literature review as part of a larger research paper (such as part of an Honors Thesis). 

Literature reviews can be written using a variety of different styles.  These may differ in the way prior research is reviewed as well as the way in which the literature review is organized.  Examples of stylistic variations in literature reviews include: 

• Summarization of prior work vs. critical evaluation. In some cases, prior research is simply described and summarized; in other cases, the writer compares, contrasts, and may even critique prior research (for example, discusses their strengths and weaknesses).
• Chronological vs. categorical and other types of organization. In some cases, the literature review begins with the oldest research and advances until it concludes with the latest research.  In other cases, research is discussed by category (such as in groupings of closely related studies) without regard for chronological order.  In yet other cases, research is discussed in terms of opposing views (such as when different research studies or researchers disagree with one another).

Overall, all literature reviews, whether they are written as a part of a larger work or as separate articles unto themselves, have a common feature: they do not present new research; rather, they provide an overview of prior research on a specific topic . 

How to Write a Literature Review

When writing a literature review, it can be helpful to rely on the following steps.  Please note that these procedures are not necessarily only for writing a literature review that becomes part of a larger article; they can also be used for writing a full-length article that is itself a literature review (although such reviews are typically more detailed and exhaustive; for more information please refer to the Further Resources section of this page).

Steps for Writing a Literature Review

1. Identify and define the topic that you will be reviewing.

The topic, which is commonly a research question (or problem) of some kind, needs to be identified and defined as clearly as possible.  You need to have an idea of what you will be reviewing in order to effectively search for references and to write a coherent summary of the research on it.  At this stage it can be helpful to write down a description of the research question, area, or topic that you will be reviewing, as well as to identify any keywords that you will be using to search for relevant research.

2. Conduct a literature search.

Use a range of keywords to search databases such as PsycINFO and any others that may contain relevant articles.  You should focus on peer-reviewed, scholarly articles.  Published books may also be helpful, but keep in mind that peer-reviewed articles are widely considered to be the “gold standard” of scientific research.  Read through titles and abstracts, select and obtain articles (that is, download, copy, or print them out), and save your searches as needed.  For more information about this step, please see the Using Databases and Finding Scholarly References section of this website.

3. Read through the research that you have found and take notes.

Absorb as much information as you can.  Read through the articles and books that you have found, and as you do, take notes.  The notes should include anything that will be helpful in advancing your own thinking about the topic and in helping you write the literature review (such as key points, ideas, or even page numbers that index key information).  Some references may turn out to be more helpful than others; you may notice patterns or striking contrasts between different sources ; and some sources may refer to yet other sources of potential interest.  This is often the most time-consuming part of the review process.  However, it is also where you get to learn about the topic in great detail.  For more details about taking notes, please see the “Reading Sources and Taking Notes” section of the Finding Scholarly References page of this website.

4. Organize your notes and thoughts; create an outline.

At this stage, you are close to writing the review itself.  However, it is often helpful to first reflect on all the reading that you have done.  What patterns stand out?  Do the different sources converge on a consensus?  Or not?  What unresolved questions still remain?  You should look over your notes (it may also be helpful to reorganize them), and as you do, to think about how you will present this research in your literature review.  Are you going to summarize or critically evaluate?  Are you going to use a chronological or other type of organizational structure?  It can also be helpful to create an outline of how your literature review will be structured.

5. Write the literature review itself and edit and revise as needed.

The final stage involves writing.  When writing, keep in mind that literature reviews are generally characterized by a summary style in which prior research is described sufficiently to explain critical findings but does not include a high level of detail (if readers want to learn about all the specific details of a study, then they can look up the references that you cite and read the original articles themselves).  However, the degree of emphasis that is given to individual studies may vary (more or less detail may be warranted depending on how critical or unique a given study was).   After you have written a first draft, you should read it carefully and then edit and revise as needed.  You may need to repeat this process more than once.  It may be helpful to have another person read through your draft(s) and provide feedback.

6. Incorporate the literature review into your research paper draft.

After the literature review is complete, you should incorporate it into your research paper (if you are writing the review as one component of a larger paper).  Depending on the stage at which your paper is at, this may involve merging your literature review into a partially complete Introduction section, writing the rest of the paper around the literature review, or other processes.

Further Tips for Writing a Literature Review

Full-length literature reviews

• Many full-length literature review articles use a three-part structure: Introduction (where the topic is identified and any trends or major problems in the literature are introduced), Body (where the studies that comprise the literature on that topic are discussed), and Discussion or Conclusion (where major patterns and points are discussed and the general state of what is known about the topic is summarized)

Literature reviews as part of a larger paper

• An “express method” of writing a literature review for a research paper is as follows: first, write a one paragraph description of each article that you read. Second, choose how you will order all the paragraphs and combine them in one document.  Third, add transitions between the paragraphs, as well as an introductory and concluding paragraph. 1
• A literature review that is part of a larger research paper typically does not have to be exhaustive. Rather, it should contain most or all of the significant studies about a research topic but not tangential or loosely related ones. 2   Generally, literature reviews should be sufficient for the reader to understand the major issues and key findings about a research topic.  You may however need to confer with your instructor or editor to determine how comprehensive you need to be.

Benefits of Literature Reviews

By summarizing prior research on a topic, literature reviews have multiple benefits.  These include:

• Literature reviews help readers understand what is known about a topic without having to find and read through multiple sources.
• Literature reviews help “set the stage” for later reading about new research on a given topic (such as if they are placed in the Introduction of a larger research paper). In other words, they provide helpful background and context.
• Literature reviews can also help the writer learn about a given topic while in the process of preparing the review itself. In the act of research and writing the literature review, the writer gains expertise on the topic .

Downloadable Resources

• How to Write APA Style Research Papers (a comprehensive guide) [ PDF ]
• Tips for Writing APA Style Research Papers (a brief summary) [ PDF ]
• Example APA Style Research Paper (for B.S. Degree – literature review) [ PDF ]

Further Resources

How-To Videos     

• Writing Research Paper Videos
• UCSD Library Psychology Research Guide: Literature Reviews

External Resources

• Developing and Writing a Literature Review from N Carolina A&T State University
• Example of a Short Literature Review from York College CUNY
• How to Write a Review of Literature from UW-Madison
• Writing a Literature Review from UC Santa Cruz  
• Pautasso, M. (2013). Ten Simple Rules for Writing a Literature Review. PLoS Computational Biology, 9 (7), e1003149. doi : 1371/journal.pcbi.1003149

1 Ashton, W. Writing a short literature review . [PDF]     

2 carver, l. (2014).  writing the research paper [workshop]. , prepared by s. c. pan for ucsd psychology.

Back to top

• Research Paper Structure
• Formatting Research Papers
• Using Databases and Finding References
• What Types of References Are Appropriate?
• Evaluating References and Taking Notes
• Citing References
• Writing Process and Revising
• Improving Scientific Writing
• Academic Integrity and Avoiding Plagiarism
• Writing Research Papers Videos

Literature reviews: Reviewing for research

• Reviewing for research
• Stand-alone review
• Scoping and planning
• Screening and appraising
• The process of reviewing
• Planning a search strategy

On this page:

“The researcher first addresses the current state of knowledge about the study question. Then, based on these findings, the researcher proposes a thesis defining an issue for further study” Lawrence A Machi, The Literature Review: Six Steps to Success

This page concentrates on undertaking a literature review as the first stage of a research-based dissertation or independent project. Such a review needs to both position your study within the wider literature and justify the research question that you are asking. This page is looking at a traditional literature review rather than a systematic review which has its own page in this guide.

What a literature review for a research project should do

Position your research within the wider body of literature on the topic.

Demonstrate an in-depth understanding of your topic area.

Identify who the major thinkers are.

Identify what research has already been done in that area.

Find gaps or new directions to help you formulate your own question.

Identify the main research methodologies in your area.

Identify the main areas of agreement or controversy.

Convince the reader that your research questions are significant , important and interesting.

Convince the reader that your research will make an original contribution.

What a literature review is, and isn't

• Critical analysis
• Evaluation of previous research on a topic
• Addressing a clearly articulated question (or questions)

• A descriptive list
• Summaries of books/articles
• Exhausted bibliography of everything written on the topic
• Your arguments and ideas

The structure of a literature review

Funnel structure.

The most common structure is one where you start your literature review looking at the bigger picture and then increasingly focus onto the specific aspects you are interested in. This is known as a funnel structure.

• As can be seen, the review starts by looking at a fair number of papers but not in great detail - these position your research  within the wider literature to show how your topic fits in to the bigger picture.
• The review then moves to consider a number of papers (less than the previous part) in a bit more detail. These papers cover more narrow categories that are closer to your topic but not matched directly. 
• The review should then focus on a few papers that are the most relevant to your work. You are likely to look at these in considerably more detail.
• You finish the review by confirming  how the literature has led you to your specific question.

Jigsaw structure

Sometimes, if your topic area has clear sub-areas it can be more appropriate to use a jigsaw structure.

• Give a proportion of your review to each sub-area.
• Discuss the links between each of the sub-areas.
• Make sure your conclusion pulls these together and shows where your research will fit into this picture.

Chronological structure

This structure is not particularly common but can be useful for some reviews - specifically when you need to show how ideas have changed through time . For example, in medicine you could look at how treatments for a particular condition have progressed from early treatments to the present day.

• You will begin with the earliest papers, grouping them together by publication date. For example, papers from 1990-1999 then papers from 2000-2009, then 2010-2019, finishing with the very latest papers from 2020 onwards. 
• You focus on how the research (ideas, theories or methods) has changed over the period and emphasise the key changes that happened.
• You finish by showing how this led you to choose the direction that your own research will take.

Top Tips Video

• << Previous: Home
• Next: Stand-alone review >>
• Last Updated: Apr 9, 2024 3:41 PM
• URL: https://libguides.hull.ac.uk/literaturereviews
• Login to LibApps
• Library websites Privacy Policy
• University of Hull privacy policy & cookies
• Website terms and conditions
• Accessibility
• Report a problem

Conducting a Literature Review: Home

What is a literature review.

A literature review is a body of text that aims to review the critical points of current knowledge on a particular topic. Most often associated with science-oriented literature, such as a thesis, the literature review usually proceeds a research proposal, methodology and results section. Its ultimate goals is to bring the reader up to date with current literature on a topic and forms that basis for another goal, such as the justification for future research in the area. (retrieved from http://en.wikipedia.org/wiki/Literature_review )

Print Resources

Many resources on research methodology include a chapter on literature review. Try these titles:

• SAGE Research Methods "SAGE Research Methods (SRM) is a research tool supported by a newly devised taxonomy that links content and methods terms. It provides the most comprehensive picture available today of research methods (quantitative, qualitative and mixed methods)across the social and behavioural sciences."

There are many resources available on the internet and in print to help you conduct a literature review. For graduate students working on a thesis, the most important resource is your graduate committee chair. Also, if you are an ILR student, don't forget to speak with one of Catherwood's reference librarians. They have subject matter expertise and can help you find research materials, as well as show you relevant databases and resources, including Zotero , an online bibliographic management system. Cornell students can also take advantage of the  John S. Knight Institute for Writing in the Disciplines , which offers a walk-in tutoring Service.

Additional Resources for Writing Literature Reviews

• Library Research at Cornell by Michael Engle Last Updated Mar 22, 2024 6499 views this year
• The Literature Review: A Few Tips on Conducting Writing Advice from the University of Toronto Health Sciences Writing Centre.
• Write a Literature Review Research guide from the University Library, UC Santa Cruz
• Learn How to Write a Review of Literature From the Writing Center at University of Wisconsin - Madison.

Searching the Library Catalog

Conducting a guided keyword search from the  Cornell University Library Catalog , using the terms "research methodology" or "qualitative research" will provide additional results, and adding the term "social sciences" to the search will help narrow the results. You can also use this list of the classification numbers for theses by department . We also offer a finding guide to dissertations and theses that you may find useful!

We also recommend you look through other theses - often your graduate chair will have copies.

• Last Updated: Feb 4, 2021 9:22 AM
• URL: https://guides.library.cornell.edu/ilrlitreview

• Open access
• Published: 11 May 2024

Effectiveness of simulation-based interventions on empathy enhancement among nursing students: a systematic literature review and meta-analysis

• Mi-Kyoung Cho 1 &
• Mi Young Kim 2  

BMC Nursing volume  23 , Article number:  319 ( 2024 ) Cite this article

Metrics details

This study aimed to secure and analyze evidence regarding the enhancement of nursing students’ empathy through simulation-based interventions. It comprehensively analyzed self-reported emotions and reactions as primary outcomes, along with the results reported by nursing students who experienced simulation-based interventions, including empathy.

This systematic literature review and meta-analysis investigated the effects of simulation-based interventions on enhancing empathy among nursing students. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines were used for the systematic review and meta-analysis. The following details were considered: population, nursing students; intervention, simulation-based interventions targeting empathy enhancement; comparators, control groups without intervention or those undergoing general non-simulation-based classes; and outcomes, self-reported empathy.

In the systematic review of 28 studies, it was found that the use of simulation-based interventions among nursing students led to an increase in empathy, albeit with a small effect size. This was demonstrated through a pooled, random-effects meta-analysis, yielding an effect size (Hedge’s g) of 0.35 (95% CI: 0.14, 0.57, p  = 0.001). The results of meta-regression and subgroup analysis significantly increased in empathy for studies published after 2019 (Hedge’s g = 0.52, 95% CI: 0.31 to 0.73, p  < 0.001), quasi-experimental research design (Hedge’s g = 0.51, 95% CI: 0.27 to 0.74, p  < 0.001), more than 60 participants (Hedge’s g = 0.31, 95% CI: 0.02 to 0.59, p  = 0.034), and simulation-based interventions in nursing education (Hedge’s g = 0.43, 95% CI: 0.22 to 0.65, p  < 0.001).

Conclusions

Considering factors such as variations in sample size, research approaches, and the effects of independent studies on empathy, this systematic literature review and meta-analysis suggests that simulation-based education can significantly improve nursing students’ overall empathy skills.

Peer Review reports

In modern society, concerns are growing regarding empathy deficits, which lead to issues such as indifference and apathy in workplace relationships—aggravating even in common social situations [ 1 ]. Empathy is a complex concept comprising an affective component of feeling and recognizing emotions from others’ perspectives and a cognitive component of understanding others’ emotions [ 2 ]. Highly empathetic professionals in health-related fields foster a high level of communication with patients, leading to positive outcomes in patient care, such as better self-care, higher patient satisfaction, and faster recovery times [ 3 , 4 ]. Although empathy is essential for healthcare workers, studies have demonstrated that it is not taught sufficiently during training in numerous fields, including medicine, nursing, dentistry, and pharmacy [ 5 , 6 ].

Empathy plays a crucial role in healthcare, as evidenced by its strong correlation with the quality of care provided to patients. When patients perceive that nurses empathize with them, they tend to feel they are receiving care tailored to their needs [ 7 ]. Therefore, improving empathy is necessary for enhancing the quality of nursing care. Efforts have been made to develop programs that foster empathy through education and training [ 8 ]. Adequate levels of empathy are essential for nursing students as they are future nurses. However, research has indicated that nursing students have lower levels of empathy than other healthcare workers do [ 9 , 10 ].

Empathy is defined as the ability to place oneself in the same position as another person and to understand and accept their position and perspective [ 11 ]. Training that enhances empathy includes simulation-based learning that recreates realistic clinical situations [ 12 ]. Additionally, healthcare can be simulated in various ways, including virtual patients, manikins, role-playing, gaming, and simulating hypothetical or disease situations [ 13 ]. Simulations in healthcare most often allow students to function in the role for which they are training, though there is evidence students’ empathy increases when they function in the role of patients because they are encouraged to understand patients’ perspectives, emotions, and experiences [ 14 ]. Whether students function in professional or patient roles during simulation, post-simulation debriefing helps students translate their simulation experiences. Post-simulation debriefing sessions have been shown to help students learn how to translate their simulation experiences into appropriate empathetic behaviors and attitudes toward patients in the real world [ 14 ]. Previous systematic reviews have included studies focusing on specific simulation methods, such as role-play or virtually simulated patients, dementia-specific virtual reality scenarios, clinical simulations with dramatization, and simulation equipment for older-adult-specific scenarios [ 15 ]. Since its emergence, improving empathy in healthcare has been the subject of several studies and meta-analyses [ 16 ]. Through a meta-analysis and effectiveness evaluation study on various simulation-based programs aimed at nursing students, both future and current medical professionals, we investigated the elements of simulation that contribute to empathy enhancement. Our study identified key elements crucial for designing effective simulation education programs, which can be reflected upon in practice. By analyzing the components of simulation-based education that impact empathy enhancement, we can identify crucial elements to enhance empathy when implementing this approach.

Simulation is becoming more prevalent as an educational approach for instilling empathy in pre-service health professional students [ 17 ]. As these various forms of simulation are applied to improve empathy, a systematic review and analysis of nursing students are needed to determine their effectiveness and the factors that should be considered. Systematic reviews and meta-analyses can consider differences in sample sizes, variations in research approaches, and the effects of interventions in independent studies while integrating the results of the included studies. Therefore, a systematic review and meta-analysis will enable an assessment of the overall effectiveness of simulation-based education in improving empathy among nursing students. This study aims to provide a foundation for simulation-based interventions by conducting a systematic literature review and meta-analysis to examine their effectiveness in improving empathy among nursing students.

Study design

This systematic literature review and meta-analysis followed the Population, Intervention, Comparison, Outcome, and Study Design (PICO-SD) framework to determine the effectiveness of simulation-based interventions in improving empathy among nursing students.

Eligibility criteria and outcome variables

This study was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [ 18 ]. This was prepared by referring to the PRISMA 2020 checklist ( https://prisma-statement.org/PRISMAStatement/Checklist.aspx , accessed May 16, 2023). In line with this study’s purpose, a systematic literature search was conducted. The inclusion criteria were as follows: the study population (P) included nursing students who received simulation training; the intervention (I) included nursing education using simulation to promote empathy; the control I group comprised those who did not receive the simulation intervention as a comparison group; and for outcomes (O), the primary outcome was empathy, while the secondary outcomes wereempathic communication, interpersonal relationships, and competency. The first post-intervention value was used to calculate the effect size. The study design (SD) involved randomized controlled trials (RCTs) and quasi-experimental studies that included manuscripts published in English or Korean from May 1971 to April 2023. Only studies that reported means, standard deviations, and concrete sample sizes were included to merge the effect sizes for the primary and secondary outcomes. The exclusion criteria were as follows: studies that included students other than nursing students, interventions that were not simulations, measured variables that were not graphically represented such that effect sizes could not be merged, studies that only presented p-values or the number of participants in each group, studies with mean and standard deviation not available, and duplicate studies. Quasi-experimental studies with a single-group pretest-posttest design were excluded.

Search strategies

Data were retrieved from eight electronic databases or e-journals, specifically PubMed, Cochrane, EMBASE, CINAHL, World of Science, SCOPUS, PQDT, and Research Information Sharing Service (RISS), for articles published in English and Korean from May 1971 to April 2023. The search protocol was registered in the PROSPERO International Prospective Register of Systematic Reviews (registration no. CRD42023423747, available at https://www.crd.york.ac.uk/prospero ) on May 16, 2023. The search formula used was Medical Subject Headings (MeSH) and text words from titles and abstracts, and the search was conducted from April 24, 2023, to June 3, 2023. The terms used in the search were (“Simulation Training”[MeSH Terms] OR “simulate*”[All Fields]) OR (“psychodrama”[MeSH Terms] OR “psychodrama*”[All Fields] OR “role-play*”[All Fields]) for interventions, and (“Empathy”[MeSH Terms] OR “empath*”[All Fields] OR “Emotional Intelligence”[MeSH Terms] OR “Emotional Intelligence”[All Fields]) for results. The data collection process for the articles included in the analysis was based on a systematic review. A literature search was conducted by two authors (MYK and MKC) with the guidance of a meta-analysis expert.

Quality assessment

The quality of the selected articles was independently assessed by two authors (MYK and MKC) using the Joanna Briggs Institute (JBI) checklist (Checklist for Randomized Controlled Trials, Checklist for Quasi-Experimental Studies [ 19 , 20 , 21 ]. In the initial quality assessment, no discrepancies were observed across most items. However, divergence arose regarding the clarity of blinding of outcome assessors to study participants. Upon thorough discussion, we agreed that a score would be assigned only if the methodology section of a study explicitly stated that outcome assessors were blinded to treatment assignment. The JBI RCT Checklist comprises the following 13 items: randomization, allocation concealment, pre-homogeneity verification, blinding (participants, interventors, and assessors), identical conditions other than experimental treatment, description of dropouts, analysis based on randomization, equivalence of outcome measures, appropriateness of outcome variable measures and statistical analysis methods, and appropriateness of the study design [ 19 ]. The JBI Quasi-Experimental Studies Checklist comprises the following nine items: certainty of cause and effect, pre-homogeneity verification, exposure to the same environment outside of the intervention, presence or absence of a control group, pre- and post-intervention effect measures, description of dropouts, equivalence of outcome measures, appropriateness of outcome variable measures, and statistical analysis methods [ 20 ]. The checklist scored “yes” as 1 and “unclear,” “no,” and “not applicable” as 0 for each item. Discrepancies in the quality assessment of the studies were resolved through consultation with a meta-analysis expert and discussions between the two authors (MYK and MKC) (Table  1 ).

Selection process

The two authors (MYK and MKC) shared the search formula, searched for data independently, and shared the bibliographic information of the articles retrieved from domestic and foreign core electronic databases and journals in an Excel file. Duplicate articles were removed by sorting by title and author using the Microsoft Excel filtering function. Based on this search strategy, relevant articles were identified through titles and abstracts, after which the full texts of the selected articles were reviewed.

Data analysis and statistical methods

The article characteristics were presented as frequencies, means, and standard deviations, and statistical analyses of effect size pooling methods were performed Z-test and p-value using MIX 2.0 Pro Ver. 2.0.1.6 (BiostatXL, Mountain View, CA, USA). As the effect sizes were continuous variables, and the number of participants in each study was small, Hedge’s g, 95% confidence intervals (CI), and the weight of each effect size were obtained using the inverse of variance [ 22 ]. The overall effect (Hedges’ g) was calculated using a pooled, random-effects model to account for between-participant variations in individual studies and heterogeneity among studies. The effect sizes indicated by Hedge’s g values of 0.15, 0.40, and 0.75 were classified as small, medium, and large effects, respectively [ 23 ]. The studies’ heterogeneity was assessed by calculating Higgin’s I 2 value, which represented the true variance or variance ratio across studies to the total observed variance. It was interpreted as heterogeneous if I 2 was greater than 50%. Subgroup and meta-regression analyses were performed to identify the sources of heterogeneity. Publication bias in the selected studies was tested using funnel plots, Begg’s test, Egger’s regression test, and the trim-and-fill method with a correction for Hedge’s g [ 24 ].

Study selection

This study followed the PRISMA guidelines during the study selection process, as illustrated in Fig.  1 . A total of 1,265 articles were retrieved from each database in Step 1. Furthermore, 578 articles were extracted by excluding duplicate studies (686) and one retracted article in Step 2, and 81 articles were extracted by excluding studies that did not fulfill the inclusion and exclusion criteria in Step 3. Finally, after a thorough review and full-text reading, 25 articles meeting our search criteria were identified for inclusion. Notably, Layton’s (1979) study was distinguished by its comparison of experimental and control groups across four distinct simulation interventions. Given the unique structure of this study, each simulation intervention was treated as a separate unit of analysis, thereby extending the total number of analyzed studies to 28. In this study, the participants of the studies included in the meta-analysis were undergraduate nursing students, and a total of 2,598 participated. The data extraction form was compiled by extracting the author, year of publication, presence or absence of IRB, number of participants, research design, experimental group’s intervention type, intervention session, session time, control group’s intervention, post-test measurement time, delayed measurement, and outcome variables. The primary variable, empathy score, and the secondary variables, empathic communication, interpersonal relationships, and competency were coded as the mean, standard deviation, and number of samples of the first post-test or the difference value of the post-pretest for both the experimental and control groups after the intervention.

PRISMA flow diagram

Study characteristics

The analysis included 28 studies, with 15 published in 2019 or later, 23 with IRB reviews before the study. The research design for simulation-based interventions included 14 RCTs: 13 with 60 or more participants, 13 simulated-based learning, 15 role-plays, and 21 studies with usual or no interventions for the control group. Simulation-based learning encompasses a variety of structured activities designed to mirror real or potential scenarios in educational settings, facilitating practice and skill development. These activities enable participants to augment their understanding, expertise, and mindset, while also providing opportunities to analyze and address realistic situations within a simulated environment [ 25 ]. Role-playing entails the enactment of specific roles within defined contexts. For instance, it may encompass a situated teaching program where patients portray themselves and articulate their experiences within a psychiatric nursing practice setting, or a role-playing training regimen conducted within an operating room situation. The intervention time or session was more than 1 h, the outcome was measured immediately after the intervention, the outcome was followed up, pre-briefing was conducted, and debriefing was conducted in study ID: 24, 11, 12, 26, 10, 6, and 14 studies. The majority of the control group adheres to a Traditional curriculum. This curriculum typically includes conventional empathic skill training through lectures, seminars, individual presentations at meetings, discussions, and similar formats. In contrast, for the experimental group, simulation involves a sequence of processes (such as orientation, pre-briefing, SP simulation performance, debriefing, and feedback). Typically, this process occurs once rather than being repeated. The impact is evaluated following the completion of this singular series of processes. The predominant empathy scale utilized was The Jefferson Scale of Empathy-Health Profession-Student (JSE-HP-S), with various other assessment tools also employed to measure empathy.

The primary outcome was empathy, which was assessed in all 28 studies. Empathic communication, interpersonal relationships, and competency were measured in study ID: 5, 6, and 9 studies, respectively (Appendix 1 ). When the sample size is small, Cohen’s d may exaggerate the effect size of an individual study. Therefore, the adjusted effect size, referred to as Hedge’s g [ 25 ], was provided along with 95% Confidence Intervals. Hedge’s g was calculated by entering the mean, standard deviation, and number of samples of each study’s experimental and control groups into the Mix Pro 2.0 program.

Risk of bias in studies

The average quality assessment score for RCTs was 8.18 (SD 0.75, range: 7–9), and the average quality assessment score for quasi-experimental studies was 8.00 (SD 1.11, range: 6–9). Among the internal validity assessment items for the RCT studies, “Q2. Was the allocation to treatment groups concealed?” for bias related to selection and allocation, and “Q5. Were those delivering the treatment blinded to the treatment assignment?” for bias related to administration of intervention or exposure, and “Were outcome assessors blind to treatment assignment?” for bias related to the assessment, detection, and measurement of the outcome were not reported in any study. Furthermore, “Q4. Were participants blinded to the treatment assignments?” was reported in only one study, and “Q12. Was an appropriate statistical analysis used?” was used to measure the validity of the statistical conclusions in three studies. Most items (Q1-5, Q7-9) that assess the quality of quasi-experimental studies have been reported. “Q6. Was the follow-up complete, and if not, were the differences between groups in terms of their follow-up adequately described and analyzed?” were reported in only seven studies (Table  1 ).

Effect of simulation-based intervention on empathy

Layton’s (1979) study was distinguished by its comparison of experimental and control groups across four distinct simulation interventions. Each simulation intervention was treated as a separate unit of analysis, thereby extending the total number of analyzed studies to 28. The effect sizes were pooled using a random-effects model and presented as Hedge’s g, 95% CI, weight, and a synthesis forest plot (Fig.  2 ). Using a simulation-based intervention among nursing students significantly increased empathy, with a total effect size of Hedge’s g = 0.35, which was a small effect based on Brydges’ criteria for interpreting effect sizes. The effect sizes indicated by Hedge’s g values of 0.15, 0.40, and 0.75 were classified as small, medium, and large effects, respectively [ 26 ]. The heterogeneity test revealed a Higgins I 2 value of 84.8%, indicating a high degree of heterogeneity among merged studies. Therefore, subgroup and meta-regression analyses were conducted for exploratory and descriptive heterogeneity analyses.

The effect of simulation-based intervention on empathy. Notes. ES: Effect size; CI: Confidence interval. Superscripts a, b, c, and d were Layton’s (1979) study divided by intervention

In the subgroup analyses, a significant increase in empathy was reported in the studies published after 2019 (Hedge’s g = 0.52, 95% CI:0.31, 0.73), IRB-approved studies (Hedge’s g = 0.39, 95% CI:0.15, 0.62), quasi-experimental studies (Hedge’s g = 0.51, 95% CI:0.27, 0.74), simulation-based interventions (Hedge’s g = 0.43, 95% CI:0.22, 0.65), and studies with no control group intervention or with usual interventions (Hedge’s g = 0.30, 95% CI:0.08, 0.53). The same was reported in studies with the intervention time per session not reported or less than 1 h (Hedge’s g = 0.42, 95% CI:0.20, 0.63), studies measuring the outcome right after the intervention (Hedge’s g = 0.38, 95% CI:0.16, 0.60), studies adopting no follow-up measurements for verifying the intervention’s long-term effects (Hedge’s g = 0.45, 95% CI:0.22, 0.68), and studies performing debriefing after simulation (Hedge’s g = 0.48, 95% CI:0.18, 0.78), compared to the studies that did not. Additionally, the effect sizes for the number of participants, pre-briefing, and quality assessment score were statistically significant (Table  2 ).

Univariate meta-regression analysis was performed to determine the potential impact of study heterogeneity on effect size, which revealed that the following variables had statistically significant effects—specifically, year of publication, IRB-approved studies, the number of participants, study design, intervention type, control group intervention, and intervention time per session (Table  3 ). The exclusion sensitivity test excluded one study from each of the 28 studies and compared the merged effect size to the original effect size to determine the impact of the estimated effect size [ 24 ]. Examining the magnitude and statistical significance of the combined effect sizes of the simulation-based interventions indicated that Hedge’s g was small, ranging from 0.31 to 42, the 95% CI (0.10 to 0.23, 0.52 to 0.61) did not include zero, and all were statistically significant. The effect size was not significantly different from that of Hedge’s g (0.35), including all 28 studies, and all studies were statistically significant. Therefore, the meta-analysis was considered robust (Table  4 ).

Effect of an intervention program on secondary outcomes

Secondary outcomes were empathic communication, interpersonal relationships, and competency, all of which were statistically significant. After the program, empathic communication with Hedge’s g = 1.35 (95% CI:0.25, 2.45), interpersonal relationship with Hedge’s g = 0.52 (95% CI:0.21, 0.84), and competency with Hedge’s g = 0.75 (95% CI:0.24, 1.26), indicating medium to large effect sizes (Table  5 ).

Publication bias

Funnel plot analysis was conducted to assess publication bias, which revealed that the individual effect sizes (blue circles) of the 28 included studies were skewed to the right, indicating some degree of publication bias (Fig.  3 ). For further analysis of publication bias, using the trim-and-fill method, the number of articles that should be added to the study was identified as nine [ 27 ]. The corrected effect size of the 37 articles was 0.04 (95%CI: -0.19, 0.26). The effect size of empathy was smaller after correction than before, but the difference was not statistically significant after correction. Moreover, the results of different methods used to detect publication bias differed. Nonetheless, the results obtained using the trim-and-fill method, which is particularly effective in illustrating publication bias in continuous variables, indicated publication bias in this study (Appendix 2 ).

Funnel plot of simulation-based interventions for empathy. Notes. Precision = 1/standard error, 0.05; limit line = 95% confidence limit

A random-effects on the results of 28 studies was performed to quantify the influence of simulation on empathy among undergraduate nursing students. The impact of the simulation-based program on empathy showed a small effect size, specifically with an effect size of 0.35. Despite variance within studies and heterogeneity in effects between studies, it was observed that the vast majority of nursing students agree that simulation increases empathy and that empathy is greater after simulation than before. The high I2 indicates significant heterogeneity, which consequently reduces the precision of summary estimates.

This aligns with previous primary research, indicating that to empathize with others beyond oneself, it’s essential to understand the other person’s perspective or position. Moreover, research suggests that such empathy can be cultivated through education [ 28 ]. This finding is also consistent with a previous study reporting that learning could improve empathy and a meta-analysis finding that empathy training improved empathy [ 17 , 29 , 30 ]. This study corroborates earlier primary research findings suggesting that empathy training ought to incorporate real-life experiences via imagination and simulations, with a focus on understanding the unobservable mental processes of others [ 31 ].

Based on a meta-regression analysis evaluating empathy [ 17 ], the factors influencing improvements in empathy are discussed below. Initially, upon scrutinizing the content of recent simulations (since 2019), they delineate as follows: Publication years after 2019 had a more significant impact on empathy than publication years before 2019. The COVID-19 pandemic has significantly influenced prelicensure nursing education, resulting in extensive disruptions that potentially affect the learning and engagement outcomes of nursing students [ 31 ]. These results reflect the diversification and sophistication of simulation education. This is because, reportedly, nursing schools in Korea have been educating and evaluating core nursing skills designated by the Korean Accreditation Board of Nursing Education as curricular and extracurricular programs to improve the clinical performance of nursing students, with an increasing number of simulation classes based on clinical scenarios similar to the clinical environment since the 2000s [ 32 ]. Since 2019, simulations have been conducted systematically and actively. Thus, the impact on empathy was significantly greater after 2019.

The causes of heterogeneity in characteristics are as follows: The effect on empathy was notably stronger with IRB approval, implying that undergoing an IRB review may signal a scientifically and ethically robust study design. Ensuring scientifically sound design and impact evaluation is crucial, even with the same program. Concerning study design, empathy’s impact was more pronounced in quasi-experimental studies compared to randomized controlled trials (RCTs). Rigorous designs, as seen in certain RCTs with multiple controls, might lead to conservative estimates of simulation effects on empathy due to tight control. Conversely, quasi-experimental studies conducted in natural learning environments suggest empathy impacts may stem from factors beyond simulation. However, further validation through research is needed. Moreover, empathy’s impact was significantly higher with 60 or more participants, likely due to increased effect power. Hence, repeated studies with sufficient participant numbers are essential for evaluating empathy improvements.

By program type, scenario-based simulations had a more significant impact on empathy than role-playing, which is in line with a previous study suggesting that role-play is usually based on a simple situation [ 33 ]. By contrast, the simulation was based on a structured scenario that allowed participants to indirectly experience the care recipient’s condition, thereby matching another person’s mind with their mental state. Moreover, role-playing has been found to contribute to empathy, as reported in a previous study in which nursing students’ critical thinking and emotional intelligence increased significantly after learning digital storytelling problem-based learning through role-playing, and a case study containing the care recipient’s disease experience and overall clinical situation [ 34 ]. More elaborate settings, assumptions, and preparations for the situation are needed to enable students to experience what being in the situation feels like rather than merely playing a role, which is expected to allow students to be more immersed cognitively and emotionally engaged with the target situation.

The intervention duration was significantly longer for four weeks or more than four weeks than for non-reported or less than four weeks, suggesting that the intervention should be at least four weeks in line with the idea that empathy is formed through continuous and steady learning [ 1 ]. This finding indicates that empathy cannot be improved through a short period of experience or training. Instead, empathy, as a process of integrating experiences and existing perceptions, is formed over time.

Other variables whose effects on empathy were not statistically significant were as follows: There were no significant differences in the time per intervention session, whether the outcome measurement time was immediate or delayed, outcome follow-up, prebriefing, debriefing, or quality score. In typical simulation training, prebriefing and debriefing are considered essential and reflective. Nevertheless, this study found no significant effect of pre-or debriefing on empathy, suggesting that the simulation context in which empathy is provided is essential, considering the nature of empathy. However, further studies on this topic are required. Furthermore, in this study, empathy was assessed using a variety of measurement tools. We also recommend that future analyses take into account the specific measurement tools employed.

The findings of this study affirm that simulation-based education, when employed across diverse clinical contexts such as women’s health, operating room scenarios, psychiatric nursing, and geriatric nursing, constitutes a fundamental approach for fostering empathy among nursing students. Among the myriad approaches aimed at enhancing empathy among medical students, the implementation of “patient simulation”—involving students in a curriculum that mirrors real patient encounters—has been noted as effective [ 35 ]. Furthermore, previous studies examining the relationship between proficiency and person-centered care competence have consistently demonstrated a positive correlation between empathy and competence in delivering person-centered care [ 36 , 37 ].

In this study, we conducted a meta-analysis of research exploring the impact of simulation-based education on empathy. Our findings indicate that simulation-based training across diverse scenarios can indeed enhance empathy levels. Specifically, focusing on immersive simulations conducted for a minimum duration of four weeks, spanning a range of authentic clinical contexts, proved to be particularly effective. Moreover, our study underscores the holistic nature of empathy, revealing its interconnectedness with other nursing competencies. As such, further research in this domain is warranted to deepen our understanding and refine instructional methodologies.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Population, Intervention, Comparison, Outcome, and Study Design

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Study Design

Randomized Controlled Trials

Research Information Sharing Service

Prospective Register of Systematic Reviews

Medical Subject Headings

Joanna Briggs Institute

Jefferson Scale of Empathy-Health Profession-Student

Shin HJ, Cho MO. Influence of critical thinking disposition and empathy ability on self- leadership of nursing students. J Korean Assn Learn Cent Curric Instr. 2023;23(2):115–27. https://doi.org/10.22251/jlcci.2023.23.2.115 .

Article   Google Scholar  

Hatfield E, Rapson R. Emotional contagion and empathy. In: The Corsini Encyclopedia of Psychology and Behavioral Science. 2009. https://doi.org/10.7551/mitpress/9780262012973.003.0003 .

Hojat M, Louis DZ, Markham FW, Wender R, Rabinowitz C, Gonnella JS. Physicians’ empathy and clinical outcomes for diabetic patients. Acad Med. 2011;86(3):359–64. https://doi.org/10.1097/acm.0b013e3182086fe1 .

Article   PubMed   Google Scholar  

Menendez ME, Chen NC, Mudgal CS, Jupiter JB, Ring D. Physician empathy as a driver of hand surgery patient satisfaction. J Hand Surg Am (Am Ed). 2015;40(9):1860–e51862. https://doi.org/10.1016/j.jhsa.2015.06.105 .

Nunes P, Williams S, Sa B, Stevenson K. A study of empathy decline in students from five health disciplines during their first year of training. Int J Med Educ. 2011;2:12–7. https://doi.org/10.5116/ijme.4d47.ddb0 .

Ferri P, Guerra E, Marcheselli L, Cunico L, Di Lorenzo R. Empathy and burnout: an analytic cross-sectional study among nurses and nursing students. Acta Biomed. 2015;86(Supplemento 2):104–15.

PubMed   Google Scholar  

Ryu HR, Bang KS. A validation study of the Korean version of the Jefferson empathy scale for health professionals for Korean nurses. J Korean Acad Nurs. 2016;46(2):207–14. https://doi.org/10.4040/jkan.2016.46.2.207 .

Ozcan CT, Oflaz F, Sutcu Cicek H. Empathy: the effects of undergraduate nursing education in Turkey. Int Nurs Rev. 2010;57(4):493–9. https://doi.org/10.1111/j.1466-7657.2010.00832.x .

Article   CAS   PubMed   Google Scholar  

Williams B, Brown T, Boyle M, McKenna L, Palermo C, Etherington J, Williams B, Brown T, Boyle M, McKenna L, Palermo C, Etherington J. Levels of empathy in undergraduate emergency health, nursing, and midwifery students: a longitudinal study. Adv Medical Educ Pract. 2014;5:299–306. https://doi.org/10.2147/AMEP.S66681 .

Williams B, Brown T, McKenna L, Boyle MJ, Palermo C, Nestel D, Brightwell R, McCall L, Russo V. Empathy levels among health professional students: a cross-sectional study at two universities in Australia. Adv Med Educ Pract. 2014;5:107–13. https://doi.org/10.2147/AMEP.S5756 .

Article   PubMed   PubMed Central   Google Scholar  

Roger D, Hudson C. The role of emotion control and emotional rumination in stress management training. Int J Stress Manag. 1995;2:119–32. https://doi.org/10.1007/BF01740298 .

Bas-Sarmiento P, Fernández-Gutiérrez M, Baena-Baños M, Correro-Bermejo A, Soler-Martins PS, de la Torre-Moyano S. Empathy training in health sciences: a systematic review. Nurse Educ Pract. 2020;44:102739. https://doi.org/10.1016/j.nepr.2020.102739 . Article Unsp 102739.

Harris KB, McCarty D, Wilson JA, Nealy KL, Waghel R, Coleman M, Battise D, Boland C. The use of a disease state simulation assignment increased students’ empathy and comfort with diabetes nutrition counseling. Curr Pharm Teach Learn. 2018;10(9):1272–9. https://doi.org/10.1016/j.cptl.2018.06.017 .

Bearman M, Palermo C, Allen LM, Williams B. Learning empathy through simulation: a systematic literature review. Simul Healthc. 2015;10(5):308–19. https://doi.org/10.1097/sih.0000000000000113 .

Eost-Telling C, Kingston P, Taylor L, Emmerson L. Ageing simulation in health and social care education: a mixed methods systematic review. J Adv Nurs. 2021;77(1):23–46. https://doi.org/10.1111/jan.14577 .

Brunero S, Cowan D, Chaniang S, Lamont S. Empathy education in post-graduate nurses: an integrative review. Nurse Educ Today. 2022;112:105338. https://doi.org/10.1016/j.nedt.2022.105338 .

Bearman M, et al. Learning empathy through simulation: a systematic literature review. Simul Healthc. 2015;105:308–19. https://doi.org/10.1097/SIH.0000000000000113 .

Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA group preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Br Med J. 2009;339:b2535. https://doi.org/10.1136/bmj.b2535 .

JBI. 2020 ( https://jbi.global/critical-appraisal-tools accessed on 16 May 2023).

Barker TH, Stone JC, Sears K, Klugar M, Tufanaru C, Leonardi-Bee J, Aromataris E, Munn Z. The revised JBI critical appraisal tool for the assessment of risk of bias for randomized controlled trials. JBI Evid Synth. 2023;21(3):494506. https://doi.org/10.11124/JBIES-22-00430 .

Tufanaru C, Munn Z, Aromataris E, Campbell J, Hopp L. Chapter 3: Systematic reviews of effectiveness. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI, 2020. https://synthesismanual.jbi.global ( https://jbi.global/critical-appraisal-tools accessed on 16 May 2023).

Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to meta-analysis. West Sussex, UK: Wiley; 2009.

Book   Google Scholar  

Sterne JA, Egger M, Moher D. Addressing Reporting Biases. In Cochrane Handbook for Systematic Reviews of Interventions: 2nd ed. Higgins JP, Green S. Eds.; Version 5.1.0 (Updated March 2011); The Cochrane Collaboration: London, UK. 2011.

Pilcher J, et al. Simulation-based learning: it’s not just for NRP. Neonatal Netw. 2012;31(5):281–8. https://doi.org/10.1891/0730-0832.31.5.281 .

Brydges CR. Effect size guidelines, sample size calculations, and statistical power in gerontology. Innov Aging. 2019;3(4):igz036.

Bown MJ, Sutton AJ. Quality control in systematic reviews and meta-analyses. Eur J Vasc Endovasc Surg. 2010;40(5):669–77. https://doi.org/10.1016/j.ejvs.2010.07.011 .

Duval S, Tweedie R. Trim and fill: a simple funnel-plot–based method of testing and adjusting for publication bias in meta‐analysis. Biometrics. 2000;56:455–63. https://doi.org/10.1111/j.0006-341X.2000.00455.x .

Jeong JO, Kim S. The effect of an empathy education program on nursing students’ empathy ability, interpersonal ability, and caring. J Korean Acad Soc Nurs Educ. 2019;25(3):344–56. https://doi.org/10.5977/jkasne.2019.25.3.344 .

Levett-Jones T, Cant R, Lapkin S. A systematic review of the effectiveness of empathy education for undergraduate nursing students. Nurse Educ Today. 2019;75:80–94. https://doi.org/10.1016/j.nedt.2019.01.006 .

Patel S, Pelletier-Bui A, Smith S, Roberts MB, Kilgannon H, Trzeciak S, Roberts BW. Curricula for empathy and compassion training in medical education: a systematic review. PLoS ONE. 2019;14:e0221412. https://doi.org/10.1371/journal.pone.0221412 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Yeo H. Predictors of empathy for nursing students. J Korea Acad-Indust coop Soc. 2017;18(1):177–84. https://doi.org/10.5762/KAIS.2017.18.1.177 .

Martin B, Kaminski-Ozturk N, O’Hara C, Smiley R. Examining the impact of the COVID-19 pandemic on burnout and stress among U.S. nurses. J Nurs Regul. 2023;14(1):4–12. https://doi.org/10.1016/S2155-8256(23)00063-7 .

Kim SY. The effect of subjective happiness, ethical sensitivity, Empathy ability on personality in nursing students. J Digit Converg. 2022;4737–45. https://doi.org/10.14400/JDC.2022.20.4.737 .

Kim KM. Effects of grit, empathy, and awareness of the nursing profession on clinical performance of nursing students. J Korean Assn Learn Cent Curric. 2022;22(16):47–58. https://doi.org/10.22251/jlcci.2022.22.16.47 .

Chang HK, Do YJ. Problem-based learning using digital storytelling: examining intelligence, critical thinking disposition, clinical competence, and metacognition. J Korean Assn Learn Cent Curric. 2021;21(8):853–66. https://doi.org/10.22251/jlcci.2021.21.8.853 .

Chen JT, LaLopa J, Dang DK. Impact of patient empathy modeling on pharmacy students caring for the underserved. Am J Pharm Educ. 2008;72(2):40. https://doi.org/10.5688/aj720240 .

Park E, Choi J. Attributes associated with person-centered care competence among undergraduate nursing students. Res Nurs Health. 2020;43(5):511–9. https://doi.org/10.1002/nur.22062 .

Download references

Acknowledgements

Not applicable.

This study was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean Government (MSIT) (No. 2022R1F1A1076248).

Author information

Authors and affiliations.

Department of Nursing Science, Chungbuk National University, 1 Chungdae-ro, Seowon-gu, Cheongju, Korea

Mi-Kyoung Cho

College of Nursing, Hanyang University, 222 Wangsimni-ro, Seongdong-gu, Seoul, South Korea

Mi Young Kim

You can also search for this author in PubMed   Google Scholar

Contributions

M.‑K.C. and M.Y.K.; data curation, M.‑K.C.; formal analysis, M.‑K.C.; investigation, M.Y.K.; methodology, M.‑K.C. and M.Y.K.; writing—M.‑K.C. and M.Y.K. All authors read and agreed to the published version of the manuscript. All authors reviewed the manuscript.

Corresponding author

Correspondence to Mi Young Kim .

Ethics declarations

Ethics approval and consent to participate.

Not applicable. This paper constitutes a literature review and does not involve human subjects; therefore, it is exempt from Institutional Review Board (IRB) review.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Cho, MK., Kim, M.Y. Effectiveness of simulation-based interventions on empathy enhancement among nursing students: a systematic literature review and meta-analysis. BMC Nurs 23 , 319 (2024). https://doi.org/10.1186/s12912-024-01944-7

Download citation

Received : 21 January 2024

Accepted : 17 April 2024

Published : 11 May 2024

DOI : https://doi.org/10.1186/s12912-024-01944-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Simulation-based interventions
• Nursing students
• meta-analysis

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]

• Do Not Sell My Personal Info

•  ⋅ 

How A Full-Funnel SEO & PR Strategy Can Drive Leads & Sales

Unlock the power of full-funnel SEO and PR integration. Learn how to influence audience behavior and achieve top search engine rankings.

Integrating digital PR & SEO with a full-funnel strategy both influences audience behavior and top-three rankings in search engines.

In my recent webinar with Search Engine Journal about how to earn links with digital PR , I didn’t have time to dive into the importance of the audience journey and a full-funnel strategy. This article is to remedy that.

Have you found that ranking in search engines is more difficult lately? Or maybe your link building isn’t driving the top 3 ranking it used to.

Google representatives have been actively discussing the decrease in the importance of links in Google’s algorithm.

What does this mean?

This doesn’t seem to mean links don’t work, but that links and brand mentions in the context of the audience’s journey are factors that help to break into the top 3 positions of Google – and, I believe, eventually, how you’ll make it into SGE or other high-visibility areas of search.

A full-funnel PR and SEO strategy is a method for building links and brand mentions in the context of the audience’s journey online, creating an off-page user experience.

Here’s a quick snapshot of the key takeaways you’ll find throughout this article:

• Audience Journey Insight: Your audience interacts with diverse content across platforms long before they search, influencing their perception and decision-making.
• Case Study: Lectric eBikes showcased how aligning content with the customer journey and securing authoritative links and mentions can significantly boost SEO performance, brand trust, and, ultimately, sales.
• Strategy Blueprint: This guide offers a structured approach for businesses to navigate digital marketing complexities efficiently, guiding the audience from awareness to conversion.
• Integration of Digital PR and SEO: Combining digital PR and SEO creates a cohesive journey that guides the audience from awareness to decision-making, influencing audience behavior and search engine rankings.
• Digital PR for SEO: Digital PR links and brand mentions are in the context of the audience journey and are needed to rank in the top three positions in search.

Off-Page Audience Experience With The Brand Matters

Google recommends being helpful to the audience and demonstrating Experience, Expertise, Authority, and trust ( E-E-A-T ) in a category to rank highly.

Your audiences are constantly looking for a solution that solves their problem or creates an opportunity, and for trusted sources to guide them through the decision-making process.

They may become aware of this problem or the solutions long before they even reach the search engines.

The chart below illustrates the sites that receive the most visits vs. those that send the most referral traffic.

This data indicates that search engines will drive the most traffic to sites, but any given audience spends a significant amount of time engaging on other platforms, digesting content about various topics.

Social and News sites get 20% and 12% of the shares of visits, respectively, while search engines drive over 70% of total traffic.

If the audience is using several touchpoints to learn about a given product or solution, wouldn’t Google use these to determine which company is a helpful source and what its E-E-A-T is for those topics?

I had my digital PR data team at PureLinq gather data for sites that ranked in Google for a seed set of cybersecurity keywords.

We then identified all of the keywords that those sites ranked in the top three positions. From that, we gathered data about the count of brand mentions and brand search volume for each domain.

The chart below shows the number of brand mentions (x-axis), U.S. search volume for brand keywords (y-axis), and the count of keywords in the top three (bubble size). Larger bubbles mean a higher count of keywords ranked in the top three.

This data set is only for cybersecurity brands.

As brand mentions increase, the bubbles grow in size, showing that a larger number of brand mentions correlates with a higher count in the top three rankings.

Brand mentions seem to influence ranking. I know there are a large number of confounding variables in this analysis. However, the idea that Google can use these measures to identify E-E-A-T is believable.

How does this play out in the real world? To understand more, I looked at Lectric eBike.

Case Study: Lectric eBike Full Funnel PR & SEO

This case study demonstrates the power of integrating digital PR with SEO to create a cohesive and effective full-funnel marketing strategy.

The key to success was the strategic alignment of content across all stages of the customer journey, ensuring that each piece contributed to building awareness, trust, and, finally, decision-making confidence.

By securing authoritative links and mentions, Lectric eBikes not only improved its SEO performance but also established itself as a trusted brand in the eyes of consumers and search engines alike.

The rise in organic visibility and brand searches directly contributed to the overall increase in website traffic and, ultimately, sales.

This case underscores the importance of a holistic approach to digital marketing, where PR and SEO work hand in hand to guide the customer journey from awareness to conversion.

Maximum Organic Visibility

At the time of this analysis, Lectric was ranked in position 1 for “ebike” and 3 for “ebikes.”

As a result of the non-brand and brand visibility, the website went from <40,000 estimated clicks per month to over 200,000 in just about 2 years – and it still seems to be growing.

How They Did It

Lectric integrated digital PR and SEO together with a full-funnel PR & SEO strategy.

Late in 2022, Lectric began to secure more media links & brand mentions.

At first glance, you might think that the growth in backlinks was the major factor in the organic traffic growth.

The famous Ahrefs chart shows a strong correlation between the number of referring domains and organic traffic growth.

However, this chart causes much confusion since it doesn’t tell the full story.

Note : Semrush and Ahrefs have very different traffic estimates lately. However, any traffic estimate will be inherently inaccurate. They estimate these based on position, average monthly search volume, and estimated clickthrough rate (CTR) using Google’s keyword planner data. They all calculate CTR differently.

A little more digging with Ahref’s content explorer and BuzzSumo shows that the site saw growth in brand mentions starting in mid-2022, around the same time the organic traffic began to grow.

Lectric gained much of the organic traffic improvement by focusing on creating content for each stage of the audience journey.

The brand generated links and brand mentions based on the audience’s needs and wants, driving brand search volume.

Lectric eBikes secured media coverage on environmental and technology news platforms and podcasts. This effort placed the brand in front of a broad audience, emphasizing its commitment to eco-friendliness and innovation.

Placements gained during this stage aimed to educate the audience on the environmental benefits of e-bikes, leveraging unique data and expert commentary to secure media coverage.

• Consumers are drawn to sustainable transportation, seeing e-bikes as a way to align with environmental values. ( Rv Lifestyle )
• The unique, self-funded growth story of Lectric eBikes appeals to those seeking reliable and ethically operated companies. ( The Verge )
• Insights from the CEO of Lectric eBikes highlight the enjoyment and utility of e-bikes, broadening the appeal to a wider audience. ( Autonocast )

These placements generated topically relevant and authoritative links and brand mentions related to the company and individual expertise.

Expert commentary and unique data appeal to journalists. Lectric could have used PR outreach to pitch Lectric’s sales data and CEO commentary to obtain media coverage.

I almost exclusively use unique data and expert commentary to secure media coverage in digital PR. This is a very effective technique.

Consideration

The company obtained mentions in articles that discussed market success, customer satisfaction, and endorsements from reputable sources. This helped build trust and positioned Lectric eBikes as a reliable choice.

Coverage included comparative discussions on pricing and quality, showing how Lectric stands out from competitors. This was crucial for audiences evaluating their options.

• Lectric eBikes’ market success and customer satisfaction offer social proof, encouraging new customers to consider an e-bike. ( Electrek article )
• Gaining trust from the audience by applying for awards from trusted sources (Arizona Tech Council).
• Media coverage and expert commentary discussing how they focus on a lower price over incentive programs ( Axios )
• Securing links and mentions in articles that evaluate options for e-bikes ( Earth911 )

Digital PR Insight

The Axios article is a great example of a technique I use in data campaigns. Find data about specific states or cities that relate to your niche. Then, use that data to pitch media about a given topic.

The website’s content was tailored to answer final questions and overcome objections. This included detailed product comparisons, reviews, and clear, transparent pricing information.

By this stage, Lectric eBikes focused on ranking for both non-brand and brand-specific keywords, facilitating easy access for users ready to make a purchase decision.

The audience is actively searching Lectric by name as a result of the digital PR.

Lectic’s site has comparison, reviews, and showroom information easily accessible in the main navigation.

It also has clear pricing and product options.

Pro tip: Brand search for a company or product seems to have an impact on a site’s ability to rank in the top three for non-brand keywords .

If you’re unfamiliar with the audience journey, here’s an overview.

What Is Full-Funnel PR & SEO?

It’s the stages of the audience journey.

A full-funnel strategy means that you create content for each stage in the audience’s journey and at the touchpoints where they are going for that type of information.

The audience journey has several stages:

• Awareness Stage: The customer becomes aware of a need, opportunity, or problem and starts looking for solutions.
• Consideration Stage: The customer evaluates different options and solutions to address their need or problem.
• Decision Stage: The customer decides on the best solution and buys or takes action.

A full-funnel strategy creates content for each stage of the audience journey.

Using e-bikes as an example:

• Situation: Consumers want to have less of a carbon footprint, and e-bikes are an option.
• Awareness: Provide graphics showing the amount of carbon an e-bike produces compared to a car.
• Considerations: Show how your e-bike compares to others in terms of carbon footprint.
• Decision: Provide information about pricing and maybe carbon savings during the checkout process.

It’s Wherever The Audience Is

The audience journey has touchpoints they use to learn or gather insights at different stages.

For example:

• Awareness Stage: The audience reads news sites, so you use digital PR to pitch unique data to journalists who cover environmental issues.
• Consideration Stage: The audience searches Google for [e-bikes with lowest carbon footprint], so you rank articles with data showing how your e-bike has the lowest carbon footprint.
• Decision Stage: The audience will search your e-bike’s brand name and purchase intent keywords, so you should focus on ranking your e-commerce pages for brand and non-brand keywords.

Targeting each stage of the audience’s journey can create topically relevant links at key touchpoints.

Designing A Full-Funnel Strategy

Designing a strategy means laying out a structured roadmap for businesses to effectively guide their target audience from awareness to decision-making.

I’m a big fan of templates that simplify processes. I created this React Workbook as part of my Digital PR for SEO Mini MBA course. You can use it to create an audience journey map, identify assets, and create an integrated PR and SEO strategy. Screenshots are below.

Create An Audience Journey Map

Start by mapping your audience’s journey, identifying key stages, touchpoints, and content needs. This will serve as the foundation for your strategy.

Understand and visualize the path your audience takes from first becoming aware of a need or problem to making a purchase decision.

The screenshot below is of an audience journey map template I created to make the process of mapping the audience journey more actionable.

Create Owned Assets For Each Stage

Develop a variety of unique assets and resources tailored to each stage of the audience journey, designed to engage, inform, and convert your target audience. These assets should be useful for both digital PR and SEO.

Instead of creating endless blog articles, create assets that are linkable and shareable.

• Expert commentary: Leverage internal experts and the C-suite to provide commentary on trends in the media.
• Unique data: Research unique data and insights into trends or data about the company’s growth.
• Expert resources: Have internal experts identify resources that are helpful to the audience.
• Guides: Create easy-to-use guides that help your audience accomplish something at a stage in their journey

In my webinar with SEJ , I gave the example of how my digital PR team earned links by analyzing publicly available data in manufacturing around the outsourcing of manufacturing overseas due to increasing labor costs in the US.

This idea was a result of recent media trends discussing the idea that US manufacturing may be in a recession. We did this for a CFO consultancy that worked with manufacturing companies.

Thus, these assets can show your overall expertise in a given category.

With assets in hand, start an integrated digital PR and SEO program.

Implement Integrated Digital PR And SEO

Launch a coordinated digital PR and SEO campaign that leverages external media coverage (links and brand mentions) and optimized online content to improve organic search visibility and drive targeted traffic to your website.

I use this integrated topic tour canvas to simplify the planning process.

• Digital PR tactics: Pitch story ideas, unique data, and expert commentary to journalists and influencers to secure media mentions and backlinks.
• SEO integration: Align your digital PR efforts with SEO by targeting keywords and phrases that have high search volumes and are relevant to the content being pitched. Use the media coverage and backlinks obtained through digital PR to enhance your website’s E-E-A-T and ranking.
• Cross-channel promotion : Amplify the reach of your media coverage and owned content through social media, email marketing, and other digital channels, ensuring that your target audience sees your brand across multiple touchpoints.
• Use digital PR strategies instead of traditional link building as it builds your E-E-A-T (Experience, Expertise, Authority, and Trust) and the associated ranking factors.
• Have a technical expert or leadership contribute to on-site content and digital PR.

Leveraging Digital PR and SEO Builds A Powerful Online Presence

Implementing this full-funnel strategy requires ongoing collaboration between your PR, SEO, and content teams to ensure all efforts are aligned and focused on guiding the audience toward conversion.

By systematically mapping the audience journey, creating engaging content for each stage, and leveraging both digital PR and SEO, businesses can build a powerful online presence by ranking in the top three and driving brand search.

More resources:

• Why Brands Should Prioritize Bottom Of Funnel Keywords In SEO
• How To Use SEO To Target Your Audience Throughout The Funnel
• The Dark Side Of Link Building

Featured Image: eamesBot/Shutterstock

Kevin is the Founder & Head of Digital PR Strategy at PureLinq, a provider of digital PR for SEO services. After ...

Subscribe To Our Newsletter.

Conquer your day with daily search marketing news.