literature review on writing skills

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PLoS Comput Biol
• v.9(7); 2013 Jul

Ten Simple Rules for Writing a Literature Review

Marco pautasso.

1 Centre for Functional and Evolutionary Ecology (CEFE), CNRS, Montpellier, France

2 Centre for Biodiversity Synthesis and Analysis (CESAB), FRB, Aix-en-Provence, France

Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications [1] . For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively [2] . Given such mountains of papers, scientists cannot be expected to examine in detail every single new paper relevant to their interests [3] . Thus, it is both advantageous and necessary to rely on regular summaries of the recent literature. Although recognition for scientists mainly comes from primary research, timely literature reviews can lead to new synthetic insights and are often widely read [4] . For such summaries to be useful, however, they need to be compiled in a professional way [5] .

When starting from scratch, reviewing the literature can require a titanic amount of work. That is why researchers who have spent their career working on a certain research issue are in a perfect position to review that literature. Some graduate schools are now offering courses in reviewing the literature, given that most research students start their project by producing an overview of what has already been done on their research issue [6] . However, it is likely that most scientists have not thought in detail about how to approach and carry out a literature review.

Reviewing the literature requires the ability to juggle multiple tasks, from finding and evaluating relevant material to synthesising information from various sources, from critical thinking to paraphrasing, evaluating, and citation skills [7] . In this contribution, I share ten simple rules I learned working on about 25 literature reviews as a PhD and postdoctoral student. Ideas and insights also come from discussions with coauthors and colleagues, as well as feedback from reviewers and editors.

Rule 1: Define a Topic and Audience

How to choose which topic to review? There are so many issues in contemporary science that you could spend a lifetime of attending conferences and reading the literature just pondering what to review. On the one hand, if you take several years to choose, several other people may have had the same idea in the meantime. On the other hand, only a well-considered topic is likely to lead to a brilliant literature review [8] . The topic must at least be:

• interesting to you (ideally, you should have come across a series of recent papers related to your line of work that call for a critical summary),
• an important aspect of the field (so that many readers will be interested in the review and there will be enough material to write it), and
• a well-defined issue (otherwise you could potentially include thousands of publications, which would make the review unhelpful).

Ideas for potential reviews may come from papers providing lists of key research questions to be answered [9] , but also from serendipitous moments during desultory reading and discussions. In addition to choosing your topic, you should also select a target audience. In many cases, the topic (e.g., web services in computational biology) will automatically define an audience (e.g., computational biologists), but that same topic may also be of interest to neighbouring fields (e.g., computer science, biology, etc.).

Rule 2: Search and Re-search the Literature

After having chosen your topic and audience, start by checking the literature and downloading relevant papers. Five pieces of advice here:

• keep track of the search items you use (so that your search can be replicated [10] ),
• keep a list of papers whose pdfs you cannot access immediately (so as to retrieve them later with alternative strategies),
• use a paper management system (e.g., Mendeley, Papers, Qiqqa, Sente),
• define early in the process some criteria for exclusion of irrelevant papers (these criteria can then be described in the review to help define its scope), and
• do not just look for research papers in the area you wish to review, but also seek previous reviews.

The chances are high that someone will already have published a literature review ( Figure 1 ), if not exactly on the issue you are planning to tackle, at least on a related topic. If there are already a few or several reviews of the literature on your issue, my advice is not to give up, but to carry on with your own literature review,

The bottom-right situation (many literature reviews but few research papers) is not just a theoretical situation; it applies, for example, to the study of the impacts of climate change on plant diseases, where there appear to be more literature reviews than research studies [33] .

• discussing in your review the approaches, limitations, and conclusions of past reviews,
• trying to find a new angle that has not been covered adequately in the previous reviews, and
• incorporating new material that has inevitably accumulated since their appearance.

When searching the literature for pertinent papers and reviews, the usual rules apply:

• be thorough,
• use different keywords and database sources (e.g., DBLP, Google Scholar, ISI Proceedings, JSTOR Search, Medline, Scopus, Web of Science), and
• look at who has cited past relevant papers and book chapters.

Rule 3: Take Notes While Reading

If you read the papers first, and only afterwards start writing the review, you will need a very good memory to remember who wrote what, and what your impressions and associations were while reading each single paper. My advice is, while reading, to start writing down interesting pieces of information, insights about how to organize the review, and thoughts on what to write. This way, by the time you have read the literature you selected, you will already have a rough draft of the review.

Of course, this draft will still need much rewriting, restructuring, and rethinking to obtain a text with a coherent argument [11] , but you will have avoided the danger posed by staring at a blank document. Be careful when taking notes to use quotation marks if you are provisionally copying verbatim from the literature. It is advisable then to reformulate such quotes with your own words in the final draft. It is important to be careful in noting the references already at this stage, so as to avoid misattributions. Using referencing software from the very beginning of your endeavour will save you time.

Rule 4: Choose the Type of Review You Wish to Write

After having taken notes while reading the literature, you will have a rough idea of the amount of material available for the review. This is probably a good time to decide whether to go for a mini- or a full review. Some journals are now favouring the publication of rather short reviews focusing on the last few years, with a limit on the number of words and citations. A mini-review is not necessarily a minor review: it may well attract more attention from busy readers, although it will inevitably simplify some issues and leave out some relevant material due to space limitations. A full review will have the advantage of more freedom to cover in detail the complexities of a particular scientific development, but may then be left in the pile of the very important papers “to be read” by readers with little time to spare for major monographs.

There is probably a continuum between mini- and full reviews. The same point applies to the dichotomy of descriptive vs. integrative reviews. While descriptive reviews focus on the methodology, findings, and interpretation of each reviewed study, integrative reviews attempt to find common ideas and concepts from the reviewed material [12] . A similar distinction exists between narrative and systematic reviews: while narrative reviews are qualitative, systematic reviews attempt to test a hypothesis based on the published evidence, which is gathered using a predefined protocol to reduce bias [13] , [14] . When systematic reviews analyse quantitative results in a quantitative way, they become meta-analyses. The choice between different review types will have to be made on a case-by-case basis, depending not just on the nature of the material found and the preferences of the target journal(s), but also on the time available to write the review and the number of coauthors [15] .

Rule 5: Keep the Review Focused, but Make It of Broad Interest

Whether your plan is to write a mini- or a full review, it is good advice to keep it focused 16 , 17 . Including material just for the sake of it can easily lead to reviews that are trying to do too many things at once. The need to keep a review focused can be problematic for interdisciplinary reviews, where the aim is to bridge the gap between fields [18] . If you are writing a review on, for example, how epidemiological approaches are used in modelling the spread of ideas, you may be inclined to include material from both parent fields, epidemiology and the study of cultural diffusion. This may be necessary to some extent, but in this case a focused review would only deal in detail with those studies at the interface between epidemiology and the spread of ideas.

While focus is an important feature of a successful review, this requirement has to be balanced with the need to make the review relevant to a broad audience. This square may be circled by discussing the wider implications of the reviewed topic for other disciplines.

Rule 6: Be Critical and Consistent

Reviewing the literature is not stamp collecting. A good review does not just summarize the literature, but discusses it critically, identifies methodological problems, and points out research gaps [19] . After having read a review of the literature, a reader should have a rough idea of:

• the major achievements in the reviewed field,
• the main areas of debate, and
• the outstanding research questions.

It is challenging to achieve a successful review on all these fronts. A solution can be to involve a set of complementary coauthors: some people are excellent at mapping what has been achieved, some others are very good at identifying dark clouds on the horizon, and some have instead a knack at predicting where solutions are going to come from. If your journal club has exactly this sort of team, then you should definitely write a review of the literature! In addition to critical thinking, a literature review needs consistency, for example in the choice of passive vs. active voice and present vs. past tense.

Rule 7: Find a Logical Structure

Like a well-baked cake, a good review has a number of telling features: it is worth the reader's time, timely, systematic, well written, focused, and critical. It also needs a good structure. With reviews, the usual subdivision of research papers into introduction, methods, results, and discussion does not work or is rarely used. However, a general introduction of the context and, toward the end, a recapitulation of the main points covered and take-home messages make sense also in the case of reviews. For systematic reviews, there is a trend towards including information about how the literature was searched (database, keywords, time limits) [20] .

How can you organize the flow of the main body of the review so that the reader will be drawn into and guided through it? It is generally helpful to draw a conceptual scheme of the review, e.g., with mind-mapping techniques. Such diagrams can help recognize a logical way to order and link the various sections of a review [21] . This is the case not just at the writing stage, but also for readers if the diagram is included in the review as a figure. A careful selection of diagrams and figures relevant to the reviewed topic can be very helpful to structure the text too [22] .

Rule 8: Make Use of Feedback

Reviews of the literature are normally peer-reviewed in the same way as research papers, and rightly so [23] . As a rule, incorporating feedback from reviewers greatly helps improve a review draft. Having read the review with a fresh mind, reviewers may spot inaccuracies, inconsistencies, and ambiguities that had not been noticed by the writers due to rereading the typescript too many times. It is however advisable to reread the draft one more time before submission, as a last-minute correction of typos, leaps, and muddled sentences may enable the reviewers to focus on providing advice on the content rather than the form.

Feedback is vital to writing a good review, and should be sought from a variety of colleagues, so as to obtain a diversity of views on the draft. This may lead in some cases to conflicting views on the merits of the paper, and on how to improve it, but such a situation is better than the absence of feedback. A diversity of feedback perspectives on a literature review can help identify where the consensus view stands in the landscape of the current scientific understanding of an issue [24] .

Rule 9: Include Your Own Relevant Research, but Be Objective

In many cases, reviewers of the literature will have published studies relevant to the review they are writing. This could create a conflict of interest: how can reviewers report objectively on their own work [25] ? Some scientists may be overly enthusiastic about what they have published, and thus risk giving too much importance to their own findings in the review. However, bias could also occur in the other direction: some scientists may be unduly dismissive of their own achievements, so that they will tend to downplay their contribution (if any) to a field when reviewing it.

In general, a review of the literature should neither be a public relations brochure nor an exercise in competitive self-denial. If a reviewer is up to the job of producing a well-organized and methodical review, which flows well and provides a service to the readership, then it should be possible to be objective in reviewing one's own relevant findings. In reviews written by multiple authors, this may be achieved by assigning the review of the results of a coauthor to different coauthors.

Rule 10: Be Up-to-Date, but Do Not Forget Older Studies

Given the progressive acceleration in the publication of scientific papers, today's reviews of the literature need awareness not just of the overall direction and achievements of a field of inquiry, but also of the latest studies, so as not to become out-of-date before they have been published. Ideally, a literature review should not identify as a major research gap an issue that has just been addressed in a series of papers in press (the same applies, of course, to older, overlooked studies (“sleeping beauties” [26] )). This implies that literature reviewers would do well to keep an eye on electronic lists of papers in press, given that it can take months before these appear in scientific databases. Some reviews declare that they have scanned the literature up to a certain point in time, but given that peer review can be a rather lengthy process, a full search for newly appeared literature at the revision stage may be worthwhile. Assessing the contribution of papers that have just appeared is particularly challenging, because there is little perspective with which to gauge their significance and impact on further research and society.

Inevitably, new papers on the reviewed topic (including independently written literature reviews) will appear from all quarters after the review has been published, so that there may soon be the need for an updated review. But this is the nature of science [27] – [32] . I wish everybody good luck with writing a review of the literature.

Acknowledgments

Many thanks to M. Barbosa, K. Dehnen-Schmutz, T. Döring, D. Fontaneto, M. Garbelotto, O. Holdenrieder, M. Jeger, D. Lonsdale, A. MacLeod, P. Mills, M. Moslonka-Lefebvre, G. Stancanelli, P. Weisberg, and X. Xu for insights and discussions, and to P. Bourne, T. Matoni, and D. Smith for helpful comments on a previous draft.

Funding Statement

This work was funded by the French Foundation for Research on Biodiversity (FRB) through its Centre for Synthesis and Analysis of Biodiversity data (CESAB), as part of the NETSEED research project. The funders had no role in the preparation of the manuscript.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved April 15, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

Writing Literature Reviews

• First Online: 11 August 2022

Cite this chapter

• Rob Dekkers 4 ,
• Lindsey Carey 5 &
• Peter Langhorne 6  

1634 Accesses

Even though there are many methods for conducting a literature review and techniques for it, as presented in Chapter 13 , the actual writing is experienced as challenging, particularly by those new to research. This calls on skills how to write, how to express thoughts of others and how to keep one’s own voice. To support these skills for the actual writing, there are points of attention, techniques and tips that can help the writing to be more effective; this is the purpose of this chapter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Aboelela SW, Larson E, Bakken S, Carasquillo O, Formicola A, Giled SA, Gebbie KM (2006) Defining interdisciplinary research: conclusions from a critical review of the literature. Health Serv Res 42(1, Part 1):329–346. https://doi.org/10.1111/j.1475-6773.2006.00621.x

Batcheller J, Kirksey KM, VanDyke Y, Armstrong ML (2012) Publish or perish: writing clinical manuscripts suitable for publication. J Contin Educ Nurs 43(1):44–48. https://doi.org/10.3928/00220124-20111003-01

Article   Google Scholar  

Bem DJ (1995) Writing a review article for psychological bulletin. Psychol Bull 118(2):172–177. https://doi.org/10.1037/0033-2909.118.2.172

von Bertalanffy L (1973) General system theory. George Braziller, New York

Google Scholar  

Camarinha-Matos LM, Afsarmanesh H (2005) Collaborative networks: a new scientific discipline. J Intell Manuf 16(4–5):439–452. https://doi.org/10.1007/s10845-005-1656-3

Crystal D (2010) A little book of language. Yale University Press, New Haven and London

Dekkers R (2017) Applied systems theory, 2nd edn. Springer, Cham

Book   Google Scholar  

Evans H (2000) Essential English for journalists, editors and writers. London, Pimlico

Kamm O (2015) Accidence will happen: the non-pedantic guide to English. London, Weidenfeld & Nicolson

Lamott A (1994) Bird by bird: some instructions on writing and life. Pantheon Books, New York

Ligon BL, Turner TL, Thammasitboon S (2017) Highlighting common pitfalls to avoid when writing the medical education manuscript. MedEdPublish 6(2):32. https://doi.org/10.15694/mep.2017.000094

Overholser JC (2011) Reading, writing, and reviewing: recommendations for scholarly manuscripts at the graduate and professional level. J Contemp Psychother 41(2):115–122. https://doi.org/10.1007/s10879-010-9163-z

Pautasso M (2013) Ten simple rules for writing a literature review. PLoS Comput Biol 9(7):e1003149. https://doi.org/10.1371/journal.pcbi.1003149

Roush K (2017) Writing your manuscript: structure and style. Am J Nurs 117(4):56–61. https://doi.org/10.1097/01.NAJ.0000515234.28924.7d

Schryen G (2015) Writing qualitative IS literature reviews—guidelines for synthesis, interpretation, and guidance of research. Commun Assoc Inf Syst 34:286–325

Truss L (2003) Eats shoots and leaves: the zero tolerance approach to punctuation. Profile Books, London

Wagner CS, Roessner JD, Bobb K, Klein JT, Boyack KW, Keyton J, Börner K (2011) Approaches to understanding and measuring interdisciplinary scientific research (IDR): a review of the literature. J Informet 5(1):14–26. https://doi.org/10.1016/j.joi.2010.06.004

Watson C (2019) Semicolon: how a misunderstood punctuation mark can improve your writing, enrich your reading and even change your life. London, 4th Estate

Webster J, Watson RT (2002) Analyzing the past to prepare for the future: writing a literature review. MIS Q 26(2):xiii–xxiii

Wellington JJ, Bathmaker A, Hunt C, McCulloch G, Sikes P (2005) Succeeding with your doctorate. Sage, London

Download references

Author information

Authors and affiliations.

University of Glasgow, Glasgow, UK

Rob Dekkers

Glasgow Caledonian University, Glasgow, UK

Lindsey Carey

Prof. Peter Langhorne

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rob Dekkers .

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Dekkers, R., Carey, L., Langhorne, P. (2022). Writing Literature Reviews. In: Making Literature Reviews Work: A Multidisciplinary Guide to Systematic Approaches. Springer, Cham. https://doi.org/10.1007/978-3-030-90025-0_15

Download citation

DOI : https://doi.org/10.1007/978-3-030-90025-0_15

Published : 11 August 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-90024-3

Online ISBN : 978-3-030-90025-0

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 04 December 2020
• Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

• Research management

‘Shrugging off failure is hard’: the $400-million grant setback that shaped the Smithsonian lead scientist’s career

Career Column 15 APR 24

Citizenship privilege harms science

Comment 15 APR 24

How I harnessed media engagement to supercharge my research career

Career Column 09 APR 24

How we landed job interviews for professorships straight out of our PhD programmes

Career Column 08 APR 24

How two PhD students overcame the odds to snag tenure-track jobs

Adopt universal standards for study adaptation to boost health, education and social-science research

Correspondence 02 APR 24

Is ChatGPT corrupting peer review? Telltale words hint at AI use

News 10 APR 24

Rwanda 30 years on: understanding the horror of genocide

Editorial 09 APR 24

Assistant Professor - Cell Physiology & Molecular Biophysics

Opportunity in the Department of Cell Physiology and Molecular Biophysics (CPMB) at Texas Tech University Health Sciences Center (TTUHSC)

Lubbock, Texas

Texas Tech University Health Sciences Center, School of Medicine

Postdoctoral Associate- Curing Brain Tumors

Houston, Texas (US)

Baylor College of Medicine (BCM)

Energy AI / Grid Modernization / Hydrogen Energy / Power Semiconductor Concentration / KENTECH College

21, Kentech-gil, Naju-si, Jeollanam-do, Republic of Korea(KR)

Korea Institute of Energy Technology

Professor in Macromolecular Chemistry

The Department of Chemistry - Ångström conducts research and education in Chemistry. The department has 260 employees and has a turnover of 290 mil...

Uppsala (Stad) (SE)

Uppsala University

Postdoctoral research fellow focused on generative modelling of synthetic cohorts in brain research

Lunds universitet, Institutionen för kliniska vetenskaper Malmö Lund University was founded in 1666 and is repeatedly ranked among the world’s top ...

Lund (Stad), Skåne (SE)

Lund University

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Open supplemental data
• Reference Manager
• Simple TEXT file

People also looked at

Original research article, developing literature review writing and citation practices through an online writing tutorial series: corpus-based evidence.

• 1 Department of Linguistics, University of Saskatchewan, Saskatoon, SK, Canada
• 2 Department of Linguistics, Stony Brook University, Stony Brook, NY, United States

Writing a literature review (LR) in English can be a daunting task for non-native English-speaking graduate students due to the complexities of this academic genre. To help graduate students raise genre awareness and develop LR writing skills, a five-unit online tutorial series was designed and implemented at a large university in Canada. The tutorial focuses on the following features of the LR genre: logical structure, academic vocabulary, syntax, as well as citation practices. Each tutorial unit includes an interactive e-book with explanations, examples, quizzes, and an individual or collaborative LR writing assignment. Twenty-nine non-native English-speaking graduate students from various institutions participated in the tutorials and completed five writing tasks. This study reports on their developmental trajectories in writing performance in terms of cohesion, lexical features, syntactic features, and citation practices as shown in three individual writing tasks. Corpus-based analyses indicate that noticeable, often non-linear, changes are observed in several features (e.g., use of connectives, range and frequency of academic vocabulary) across the participants' writing samples. Meanwhile, citation analysis shows a steady increase in the use of integral citations in the participants' writing samples, as measured with occurrence by the number of sentences, along with a more diverse use of reporting verbs and hedges in their final writing samples. Pedagogical implications are discussed.

1. Introduction

Well-developed academic writing skills in English are an indispensable prerequisite for academic success of graduate students since these skills are commonly expected in thesis writing and academic publication. Even after passing the entry English language requirements, many non-native English-speaking international students still experience multiple difficulties with academic writing ( Cheng and Fox, 2008 ; Zhang, 2011 ). Most studies of writing address either the contexts of standard English proficiency tests, such as TOEFL ( Riazi, 2016 ), or undergraduate academic writing courses (e.g., Dafouz, 2020 ). By contrast, very little is known about the specifics of writing skills development by graduate students (e.g., Shi and Dong, 2015 ). The available studies are limited in their scope, disciplines, number of participants, ethnic backgrounds ( Cheng, 2007 ), or methods involving opinions rather than actual writing features ( Zhang, 2011 ). To the best of our knowledge, there are no studies that consider the potential of extracurricular tutorials to impact the writing proficiency of graduate students.

Our focus on the genre of “literature review” is motivated by its high frequency among the types of graduate writing assignments across multiple disciplines in Canadian graduate programs ( Shi and Dong, 2015 ). “Literature review” is defined as a survey of “the relevant literature to discuss the state of knowledge or identify gaps in research” ( Shi and Dong, 2015 : 131). Literature review has been one of the challenges faced by graduate students, native and non-native-English speaking ones alike ( Chen et al., 2016 ; Badenhorst, 2018 ). Graduate-level international students often come to Canadian universities unaware of the specific requirements of this genre and experience multiple difficulties with it, such as avoiding plagiarism, building coherent narratives, citation formats, the use of vocabulary, and grammar forms, which impede their academic progress and cause other negative consequences ( Shi and Dong, 2015 ). While various efforts have been made in English for academic purposes (EAP) programs to help graduate students develop academic writing skills and/or genre awareness (e.g., Storch and Tapper, 2009 ; Li et al., 2020 ; Crosthwaite et al., 2021 ), programs or courses dedicated to literature review writing are still sparse and under-investigated in research.

From the theoretical perspective, this study is motivated by exploring the specific linguistic structure and textual parameters associated with successful academic writing in graduate context. In general, the major domains of writing success (or failure) have been reported to include lexis (including academic vocabulary), syntax (at phrasal and sentence levels), cohesion, and citation/referencing conventions ( Biber et al., 2011 ; Mansourizadeh and Ahmad, 2011 ; Mazgutova and Kormos, 2015 ). On the practical side, our project is aimed at identifying a cost-effective way to develop graduate students' writing skills with minimal pressure on the faculty and university resources. In Canada, not all universities can yet provide adequate support for the academic writing needs of international students at the graduate level ( Okuda and Anderson, 2018 ). As a potential solution, we developed an extracurricular writing tutorial series with free access to an online learning platform. In this paper, we are exploring whether the developed tutorial delivers practical outcomes in terms of improving academic writing proficiency in the genre of Literature review.

Our study, therefore, answers a strong demand for additional means of assisting graduate students with their academic writing skills development in the context of Literature review genre. We consider the feasibility and potential benefits of online academic writing tutorials for graduate students' writing performance in terms of linguistic development and citation practices. The materials of the study come from writing samples collected from our online writing tutorial series on literature review writing.

2. Literature review

2.1. learners' development of academic writing.

While we acknowledge the role of “history, ideology, and socio-cultural structures” in writing skills development ( Zhang, 2011 , p. 41), in this particular paper, we treat writing from an “autonomous” perspective of the academic literacies model which sees writing as a development of specific skills and genre repertoires ( Lea and Street, 1998 ). More specifically, we adopt the combination of “the linguistic and genre” approach focusing mostly on linguistic structures and writing within a specific genre ( Xu, 2019 ). For the genre of literature review, it is not surprising to see studies applying genre analysis to graduate students' writing samples ( Flowerdew and Forest, 2011 ) or focusing on writer stances ( Shahsavar and Kourepaz, 2020 ). However, specific linguistic features in this genre, especially by graduate students, have not been extensively researched. When it comes to gauging learners' development of academic writing in general, linguistic measures are often operationalized at various levels such as lexical and syntactic complexities ( Bulté and Housen, 2014 ). At the lexical level, measures of complexity and sophistication have been employed in several empirical studies. For example, in Mazgutova and Kormos (2015) pretest-posttest study, changes in five lexical measures, i.e., two diversity measures, two-word frequency measures, and one semantic similarity measure, were investigated in the argumentative writing samples by two groups of students in a 4-week intensive EAP program. While the two groups enrolled in the same EAP program as two cohorts, it was found that at the end of the program, the L2 students in one group were able to gain significant improvement in all five lexical measures, while the students in the other group showed progress in only one diversity measure (squared verb variation) and a frequency measure of AWL (Academic Word List) words. With their corpus tools named the Tool for the Automatic Analysis of Lexical Sophistication (TAALES), Kyle and Crossley (2015) added a series of new measures (e.g., word or n-gram frequency and range indices with various corpora references, academic vocabulary indices, and word information indices) to represent lexical sophistication, which is also related to lexical diversity and difficulty of lexical items. Through examining the relationships between the lexical sophistication indices yielded by TAALES and the holistic scores of TOEFL writing samples from two writing task types, Kyle and Crossley confirmed that several features were significantly correlated with the writing holistic scores. Nevertheless, task effect in Kyle and Crossley (2015) seemed noticeable as a set of four lexical sophistication features (i.e., all word range and bigram frequency logarithm referenced to BNC written corpus, number of hypernyms, and word information) were able to explain over 30% of the variance in the holistic scores of TOEFL independent writing task, while only two features (e.g., number of noun hypernyms and a genre-based range index) were retained in the final regression model to explain <10% of score variances for the TOEFL integrated writing task. Since TOEFL integrated writing only bears some resemblance to source-based academic writing, it is worth investigating what kind of lexical features may exhibit progressive trends in the genre of literature review as sampled in our online tutorial series.

Lexical features are also closely related to cohesion quality, an important aspect of writing performance. For example, the use of connectives and lexical overlaps across sentences or paragraphs can contribute to local cohesion or global cohesion ( Crossley et al., 2019 ). However, previous studies on L2 student essays have established that cohesion features may have different effects on writing performance, often with local cohesion features, such as sentence-level lexical overlaps and use of conjunctions, associated negatively with writing scores ( Crossley et al., 2016 ). It is less clear whether similar patterns can be observed in literature review samples written by L2 graduate students.

For syntactic features in academic writing, phrasal complexity has attracted much attention because features like various phrasal embedding are found to be more distinctive than clause-level features in academic writing ( Biber et al., 2011 ). This trend was confirmed by Kyle (2016) in a series of analyses with his corpus tool named Tool for Automatic Analysis of Syntactic Sophistication and Complexity (TAASSC). One goal of Kyle (2016) was to identify a group of syntactic features that are associated with writing proficiency as measured with TOEFL independent writing tasks. With 13 syntactic features, Kyle's final regression model was able to explain 34.2% of variances of TOEFL holistic scores. Six of the features were phrasal complexity indices (i.e., counts and standard deviation of dependents per nominal, prepositions per non-pronoun nominal, dependents per non-pronoun object of a preposition, counts, and standard deviation of dependents per direct object), contributing to an explanation of 17.1% of variances. In other words, the essays written by high-proficiency test takers are more likely to show higher phrasal complexity strengthened by more dependents for noun phrases that function as objects. The other features included five indices of syntactic sophistication that are related to the association strengths or frequencies of construction, and two clausal complexity indices that conceptualize the counts and types of clausal dependents.

Differences in phrasal complexity have also been noted in comparative studies of the writings of novice writers and expert writers. For example, following Biber et al. (2011) noun phrase development model, Ansarifar et al. (2018) compared the phrasal complexity features in the abstracts of research articles written by Persian L1 graduate students (MA and Ph.D. students) and experts in applied linguistics. Their analysis results indicate that four complex noun phrase features, i.e., nouns as pre-modifiers, -ed participle as post-modifiers, attributive adjectives/nouns as pre-modifiers, and multiple preposition phrases as post-modifiers, were strongly associated with the level of expertise as more experienced writers tended to use highly modified noun phrases.

Somewhat different from the general findings in cross-sectional studies on linguistic complexity, Kyle (2016) longitudinal analyses of two small corpora of English language learners' writing samples show that while some common trends were observed as with those in the cross-proficiency comparison, the learners' progress was mainly manifested in clausal complexity such as mean length of t-units and complex nominals per clause, as well as syntactic sophistication indices such as verb-VAC (verb-argument construction) frequency. These differences may be related to the nature of writing tasks (untimed vs. timed tasks), genres, as well as educational levels of the writers.

2.2. Citation practices in academic writing

Academic writing, especially the section of literature review, by nature, requires citing other studies to make an argument and to engage in academic conversations. Citation practices have been studied from the following perspectives: Citation forms or types, citation functions, and reporting verbs ( Hyland, 1999 ; Swales and Feak, 2012 ).

Citation forms or the surface form of citations are often distinguished as integral and non-integral in-text citations based on whether a cited work functions syntactically as a part of a sentence ( Swales and Feak, 2012 ). Both expertise level and nativeness seem to affect writers' citation practices. For example, Mansourizadeh and Ahmad (2011) noted that at a Malaysian university, the research papers in chemical engineering produced by non-native English-speaking expert writers showed a higher density (normalized frequency per 1,000 words) of citations in each section of the research papers than the papers written by non-native novice writers (students) in the same field (e.g., 23.19 vs. 20.1 citations per 1,000 words in the section of Introduction). In addition, in terms of citation types, non-integral citations constitute the majority of the citations (86.47% in experts' papers and 73.23% in novice writers' papers, respectively). However, expert writers demonstrated a balanced use of the two forms of integral citations, namely, verb-controlling where a cited work is introduced by a verb and naming forms which include a cited work as a part of a noun phrase, whereas novice writers heavily relied on the former. With regard to citation functions, while both writer groups have utilized citations for different purposes, expert writers seemed to have more evenly used four main functions while almost half of the citations in novice writers' papers fulfilled the function of attribution. In a study on 100 source-based papers by first-year university ESL students, Lee et al. (2018) revealed a lower citation density (about 10 citations per 1,000 words) and a more balanced use of integral and non-integral citations (53 vs. 47%). This pattern may be attributed to the design of the first-year writing class at the university, where students may be taught to use both citation forms.

Likewise, the use of reporting verbs may vary across expertise levels as well as disciplines both in terms of frequency and types ( Hyland, 1999 ). In a comparison of a learner corpus of academic essays written by first-year undergraduate students and a reference corpus of published research articles in the same disciplines, Liardét and Black (2019 ) revealed that novice writers on average used fewer reporting verbs than expert writers. This finding is supported by Marti et al. (2019) comparative study on reporting verbs. Furthermore, as shown in Liardét and Black (2019 ), compared to the reporting verbs in learners' essays, a larger proportion of the reporting verbs used by experts are used to convey writers' evaluative stance. Related to stance-making in citations is the use of boosters and hedges. Aull and Lancaster (2014) corpus-based comparative analysis indicates that the frequencies of hedges and boosters were affected by writers' expertise levels. More specifically, first-year university students' argumentative essays tend to use fewer hedges and more boosters than the ones written by upper-level students and experts. It is not clear whether similar patterns can be observed in L2 graduate students' literature reviews.

As discussed above, novice writers, including L2 graduate students, need to pay attention to citation practices for better academic socialization. Nevertheless, only limited studies have focused on L2 learners' development of citation practices in academic writing.

2.3. Research questions

Considering the importance of linguistic performance and citation practices in academic writing, as well as the lack of empirical studies on L2 learners' development of literature review writing, this study aims to answer two research questions in the context of an online tutorial series of literature review writing:

1. What linguistic features may change in L2 graduate students' literature review texts over the course of an online academic writing tutorial series?

2. How do L2 graduate students' citation practices change in terms of citation forms and stance features (reporting verbs, hedges, and boosters) over the course of an online academic writing tutorial series?

3. Research methodology

3.1. the writing tutorial series.

To help address some challenges faced by ESL graduate students in writing literature reviews, we developed a 5-unit online tutorial series with a focus on the following aspects of literature review writing: genre requirements, logic and structure in literature review, sentence structures, academic vocabulary, and grammar of reported speech (see Table 1 ). The selection of the themes was made based on the genre features of literature review ( Flowerdew and Forest, 2011 ) as well as the typical challenges in academic writing ( Chen et al., 2016 ). Each unit consists of a core tutorial explaining key concepts with both positive and negative examples, a writing task with peer review or group writing activity, and supplementary activities such as awareness-raising activities for comprehension check, discussion forum for participants to share writing experiences and ideas, and a wrap-up quiz.

Table 1 . Overall structure of the writing tutorial series.

The tutorial series is delivered via MoodleCloud, a cloud-based learning management system, as self-paced sessions but with fixed due dates for writing tasks. Each core tutorial is presented in the format of H5P (HTML5 Package) interactive e-book (see Figure 1 ), which allows participants to browse the content and complete awareness-raising activities embedded in the e-book. The total duration of the tutorial series is 3.5 months. The writing tasks require the participants to write a literature review on a given topic. The expected length is 700–800 words, excluding the references section. Three out of five writing tasks are to be completed individually with a round of online peer review whereas the other two tasks are collaborative tasks contributed by two to three participants. The individual writing tasks were evenly arranged in this tutorial series, with one at the beginning, one in the middle (Unit 3), and one at the end. Such a distribution allows us to study participants' writing development without direct input from their peers as in collaborative writing tasks. A bibliography on each topic is provided to the participants as a starting point for literature search and synthesis. Since this tutorial series was fully online and primarily managed by a non-teaching administrator, there was no restriction regarding the resources that a participant could access to help prepare the writing tasks. This study focuses on individual writing tasks only to track participants' writing development throughout the online tutorial series.

Figure 1 . Screenshot of H5P interactive e-book in tutorial 1.

3.2. Participants

To reach out to the targeted participants, we disseminated recruitment flyers digitally via university newsletters, online announcement, and emails to university faculty members. We also employed snowball sampling strategy to encourage the registered participants to share the project information with their peers. Despite strong interest from prospective participants, only 29 participants have completed the tutorial series and the corresponding writing tasks, partly due to the impact of the COVID-19 pandemic. There were 20 female participants, seven male participants, and two did not provide gender information. Based on their responses to a background survey, 19 participants were current master's students, four were doctoral students, and six were prospective graduate students without clear education level information. The main study fields of the participants include Linguistics (7), Education (6), Political Science (5), Computer Science (2), Veterinary Science (2), Health Science (2), Humanities (1), Agriculture (1), and Business (1). The first languages of the participants included Chinese (4), Farsi (4), Russian (2), Bengali (2), Hindi (2), Punjabi, Turkish, Vietnamese, Urdu, Spanish, Igbo, Luo, Portuguese, Czech, Jamaican Patois, Lusaka, Arabic, and Ukrainian. The average score of their most recent standardized English language tests is about IELTS 7 or equivalent after a score conversion from other tests, with a range from IELTS 6.5 to 8.5 and a standard deviation of 0.55.

3.3. Data collection and analysis

After a brief online orientation via email, the participants started self-paced study of tutorial content and prepared for the writing task accordingly. The duration of a tutorial unit was 3 weeks when its writing task is designed to be completed individually by the participants. In the case of collaborative writing tasks, participants had 2 weeks to complete the unit (Units 2 and 4). The writing samples were rated by three raters based on a 10-point analytical rating scale at the end of each unit. The rating scale covers ten aspects of the writing quality, such as quality of selected source materials, integration of source materials, overall structure, clarity of ideas, grammatical accuracy, coherence and cohesion, and vocabulary quality. Written feedback, along with the ratings, from the raters was shared with the participants. Considering that the rating practices in this project focused more on written feedback and this paper primarily employed a more corpus-based approach, we chose not to include the ratings in data analyses. The literature reviews written to three individual writing tasks (Tasks 1, 3, and 5) were collected and then processed to retain the body of the literature review only for text analysis (see Table 2 ). Overall, the average length of the writing samples is 733 words. Nevertheless, a possible topic effect is reflected in average essay lengths, with Task 1 ( Online learning: Pros and cons ) elicited relatively longer responses (782 words and 42 sentences on average) and Task 5 (Pacifism, peace-making, just/justifiable war) shorter responses (702 words and 33 sentences on average), while average sentence lengths are mostly comparable across the topics.

Table 2 . Participant writing sample corpus (excluding references sections).

The corpus data were analyzed with three open-source tools developed by Kyle et al. (available at www.linguisticanalysistools.org/ ) for the linguistic features related to text cohesion (Tool for the Automatic Analysis of Cohesion or TAACO 2.0, Crossley et al., 2019 ), syntactic features (Tool for Automatic Analysis of Syntactic Sophistication and Complexity or TAASSC 1.3.8, Kyle, 2016 ), and lexical features (Tool for Automatic Analysis of Lexical Sophistication or TAALES 2.2, Kyle and Crossley, 2015 ). While these three analytical tools produce a large number of features or indices, we selected a small portion of the normalized features based on their relevance to the quality of academic writing in general ( Crossley et al., 2016 ; Kyle and Crossley, 2016 , 2017 ). For example, While TAACO yields 194 features in seven categories, such as Type token ratio (TTR) and density, sentence-level lexical overlap, paragraph-level overlap, semantic overlap, connectives, givenness, and source text similarity, four categories are either overlapped with the output from another tool (e.g., TTR from TAALES) or less relevant to the writing genre in this study (e.g., givenness and source text similarity). As a result, only 41 out of 194 features from TAACO in three categories were selected partially based on the findings mentioned in Crossley et al. (2019) . Section 4.1 provides the details of the selected features.

A Python script was developed to automatically extract in-text citation instances for further analyses of citation types (i.e., integral vs. non-integral citations) and to identify reporting verbs, boosters, and hedges used in those instances based on corresponding word lists A list of 122 reporting verbs was prepared based on Liardét and Black (2019) ; a list of 56 hedges and a list of 38 boosters were adapted from Aull and Lancaster (2014) .

Considering the common patterns of non-normal distribution of frequency or ratio-based linguistic features ( McEnery and Hardie, 2012 ), we employed the Friedman test, a non-parametric statistical test, which is similar to repeated measures ANOVA, to compare the selected features across the three individual writing tasks to gauge participants' development in academic writing. Nemenyi post-hoc test was used for pairwise comparisons when a Friedman test detected differences of statistical significance. We understand that using Friedman tests multiple times on the data tends to introduce Type I errors and we decided to choose a more stringent p-value ( p ≤ 0.001) to minimize such influence. Python packages SciPy1.8.1 ( Virtanen et al., 2020 ) and scikit-posthocs ( Terpilowski, 2019 ) were used for statistical analyses.

4.1. Research question 1: Linguistic features

Regarding the lexical features, we narrowed down to nine features that are related to academic writing from the output of the software TAALES: six range or frequency features in content words, bigrams, and trigrams with reference to the academic register in COCA (Corpus of Contemporary American English), two percentage features related to Academic Word List (AWL) and Academic Formulas List (AFL), and academic lemma TTR (Type-Token Ratio, ratios of unique lemma count to token count). The results of Friedman tests indicate that two features exhibited at least one statistically significant difference among the writing samples from the three tasks (see Table 3 ): frequency of trigrams in COCA Academic (chi-squared = 13.52, p = 0.001, Kendall's W = 0.233) and proportion of AWL (chi-squared = 20.07, p < 0.001, Kendall's W = 0.346). Meanwhile, the differences in other three features were close to statistical significance level: frequency of content words in COCA Academic (chi-squared = 10.21, p = 0.006), bigram range in COCA Academic (chi-squared = 6.69, p = 0.035), proportion of core AFL (chi-squared = 9.93, p = 0.007), and academic lemma TTR (chi-squared = 9.17, p = 0.010).

Table 3 . Lexical features across three writing tasks.

The first feature in Table 3 “COCA academic frequency content words” reflects the level of frequency scores of the content words that appear in the academic register or sub-corpus in COCA, with a higher value suggesting more frequent content words. Follow-up pairwise comparisons confirm that the content word frequency in Task 1 was significantly lower than in Task 3 ( p = 0.023) and Task 5 ( p = 0.011), while no statistically significant difference exists between content word frequency in Tasks 3 and 5. The feature of the bigram range in COCA Academic shows the distribution range of academic bigrams in COCA Academic. A significant difference in this feature is noted between the texts in Tasks 3 and 5 ( p = 0.033), with the latter having a larger value in bigram range. The third COCA Academic-reference feature is the frequency of trigrams. The participants' writing exhibited a decreasing trend in this feature from Task 1 to Task 3 ( p = 0.001), meaning that less frequent trigrams were used in the texts for Task 3. There is no statistically significant difference in this feature between the texts for Tasks 3 and 5.

The texts also differed in the proportions of words from AWL in Task 1 and the other two tasks (Task 1, p = 0.001 and Task 5, p = 0.005), with the texts in Task 1 having a higher proportion of AWL words. For the percentage feature related to AFL, texts in Task 1 had a higher proportion of AFL multi-word units than those in Task 3 ( p = 0.005) while Tasks 3 and 5 had similarly lower proportions. On the other hand, the texts in Task 1 had a lower TTR of academic lemma than those in Task 3 ( p = 0.010), while Tasks 3 and 5 had a similar level of academic lemma TTRs.

For these lexical features, different developmental patterns are observed. Two features (i.e., content words frequency and academic lemma TTR) showed increasing trajectories and two (i.e., trigram frequency and Core AFL percentage) had somewhat downward trends. The other two features remained similar values in Tasks 1 and 5, but differed more remarkably in Task 3.

To explore participants' development of academic writing in terms of text cohesion, we focused on 41 cohesion features in three categories from TAACO, namely, 12 features related to lexical overlaps across sentences, 4 features related to semantic similarity, and 25 connective-related features. These features were compared across the three tasks using the Friedman test. While 14 features were found to be statistically significant at alpha = 0.05 level, only three were retained and reported in Table 4 because some of the features were either highly correlated and/or were extremely low in value. For example, the correlation between the feature word2vec-based semantic similarity across a 2-sentence span and word2vec-based semantic similarity across a 1-sentence span is 0.77, which shows a strong relationship with probably some overlaps in their assessed constructs.

Table 4 . Cohesion features across three writing tasks.

Table 4 shows the mean and standard deviation of the three cohesion features that exhibited noticeable differences across the three writing tasks. The feature “adjacent overlaps of content words” captures local cohesion achieved through repeating content words or lemmas (e.g., lexical verb, noun, adjective) in a target sentence and the following two sentences (chi-squared = 7.103, p = 0.029). A higher value suggests more occurrences of repeated content words (lemmas) divided by type count in the same text. Nemenyi post-hoc tests for pairwise comparison indicate a significant difference in this feature between Task 1 and Task 5 ( p = 0.023), with texts for Task 5 showing a lower level of adjacent overlaps in content words but a larger standard deviation.

Different from the lemma overlap-based measure of cohesion, most semantic similarity measures in TAACO take advantage of statistical representation of word meanings as used in latent semantic analysis or LSA, latent Dirichlet allocation or LDA, and word2vec, respectively to numerically evaluate the semantic distance between text blobs. In this study, the difference in the word2vec-based semantic similarity across a 2-sentence span shows statistical significance across the three tasks (chi-squared = 43.655, p < 0.001, Kendall's W = 0.753). The follow-up pairwise comparisons confirm that the texts for Task 5 exhibited stronger semantic similarity than the ones written for Task 1 ( p = 0.001) and Task 3 ( p = 0.001). The use of basic connectives (e.g., for, and, nor ) also distinguishes the writing samples (chi-squared = 7.103, p = 0.029) to some extent with more basic connectives used in the Task 5 texts than in the Task 1 texts ( p = 0.023) and no significant difference between Task 1 and Task 3.

Overall, the patterns of cohesion features observed in this study suggest that the participants developed better local cohesion through the use of connectives and also by higher sentence-level semantic similarity, but with fewer incidences of repetition of content words across sentences.

From the output of TAALES for syntactic feature analysis, we started with 25 distinguishing features that are relevant to academic writing, including 20 noun phrase complexity features and five clause-complexity features. Four features showed statistically significant differences across the three writing tasks (see Table 5 ).

Table 5 . Syntactic features of statistical significance across three writing tasks.

The feature “average number of dependents per nominal phrase” saw a decreasing trend from Task 1 to Task 5 (chi-squared = 13.24, p = 0.001, Kendall's W = 0.228), with the texts for Task 1 being significantly higher in this feature than those for the other two tasks ( p = 0.005, p = 0.005). For this noun-phrase complexity feature, the dependents could be modifiers fulfilled by an adjective, noun, clause, and preposition phrase. The other three distinguishing features belong to clause-complexity: number of clausal complements per clause (chi-squared = 6.90, p = 0.032), coordinate phrases per clause (chi-squared = 6.28, p = 0.043), and complexity nominal per clause (chi-squared = 7.66, p = 0.022). For the last two clause complexity features, the main difference in the ratio of coordinate phrases per clause exists between Tasks 1 and 3 only ( p = 0.066). The difference in complex nominal per clause between Tasks 1 and 5 is statistically significant ( p = 0.016).

Overall, two of the significant features showed some decrease in both phrasal complexity and clausal complexity in the final writing task. The other two features exhibit a drip and a peak in Task 3, respectively. In other words, the development of syntactic features is not linear as shown in the writing samples.

4.2. Research question 2: Citation practices

The second question concerns participants' citation practices in terms of citation types and stance features in citation instances. Table 6 presents the descriptive statistics of both raw counts and normalized counts of integral and non-integral citations used by the participants in three writing tasks. The results of the Friedman tests indicate that there are no statistically significant differences in citation types across the three tasks. Overall, non-integral citations were dominant in participants' writing in this study. For example, the average raw count of non-integral citation is about 11 whereas that of integral citations is close to 2. The normalized counts of citation types also show similar trends. Similar to the distribution pattern of some linguistic features discussed above, the texts for Task 3 had higher raw counts of citation instances, compared to the other two tasks. This again points to the possible topic effect on writing performance.

Table 6 . Citation practices across three writing tasks.

As for the stances in citation instances, there were no statistically significant differences in the frequencies of occurrence of reporting verbs, boosters, and hedges across the writing tasks while increasing trends are evident for these features, especially the normalized frequencies (see Table 7 ). For example, the texts written for Task 5 employed slightly more reporting verbs, hedges, and boosters than the first two individual writing tasks.

Table 7 . Reporting verbs, hedges, and boosters in citation instances across three writing tasks.

A closer look at the specific instances of reporting verbs, hedges, and boosters reveals that all three features have noticeable differences in their type-token ratios or TTR, especially from Task 1 to the other tasks. For example, in Task 1, a total of 93 unique reporting verbs appeared in the texts. With many of the reporting verbs repeated in the texts, the count of reporting verbs is 225. Therefore, the TTR of reporting verbs in the Task 1 texts is 0.41 (93 types/225 tokens). By contrast, the TTRs for Task 3 and Task 5 are 0.48 (119/246) and 0.46 (118/255), respectively, suggesting higher diversity in participants' use of reporting verbs along with more frequent use of reporting verbs. Likewise, the TTR of hedges increases from Task 1 (24/257, 0.093) to Task 3 (30/313, 0.096) and Task 5 (29/223, 0.130). For the specific cases of boosters, the texts in Task 1 have a TTR of 0.24 (17/79), whereas Tasks 3 and 5 saw higher frequencies of occurrence of boosters (111 and 110, respectively) but with the same count of types (20) and therefore lower TTR values (0.18). This is mainly because of the relatively small list of boosters (38) used in this study for booster extraction. Since we only calculated the TTRs of these features at task level to explore possible differences, we did not run inferential statistics.

Table 8 lists the top 10 words in each category across three writing tasks. Each category shares several common items across three writing tasks with some minor differences emerging. For example, “suggest”, “conduct”, “demonstrate”, and “identify” are unique among the top 10 reporting verbs in Task 1. While it is difficult to pinpoint possible causes of these differences, we speculate that they may be linked to the topic effects as the sources cited for different topics may differ in terms of perceived certainty and acceptance of findings, which in turn may affect writers' stance-making decisions.

Table 8 . Top 10 reporting verbs, hedges, and boosters in citation instances across three writing tasks.

5. Discussions and conclusions

As shown above, the development of literature review writing was not clear-cut over the course of the online tutorial series. On the one hand, participants' writing displays progress in local cohesion quality and some academic lexical features from the first individual task to the last one (Task 5). However, their writing samples also see some decreases in syntactic features and some academic lexical features, which can be an indication of the participants' adjustment of sentence variety in response to one of the most frequent comments from raters on long sentences and the grammatical issues found in them. The participants' citation practices remained at similar levels in terms of citation types and stances manifested through reporting verbs, hedges, and boosters. However, they seem to have more stance-making resources at their disposal as evidenced by more types of reporting verbs, hedges, and boosters. It should be noted that the increase or decrease in values does not necessarily suggest improvement in academic writing yet. These observations are to be discussed in light of the relevant literature.

5.1. Linguistic features

If we consider the complex nature of language learning in general, the diverse patterns in the changes of linguistic features from Task 1 to Task 5 are less surprising as linear progressions in academic writing are rarely achieved. For example, in a longitudinal case study of an L2 graduate student at an Australian university, Rosmawati (2014) tracked the development of complexity and accuracy in the student's argumentative essays over a semester. It was found that a high level of variability existed for both complexity and accuracy development, supporting the non-linear and dynamic nature of L2 learning. Furthermore, a clear difference in the interactions between complexity and accuracy measures was revealed at different measurement points or stages in the semester.

Non-linear patterns are obvious with the lexical features reported in this study as three out of six features (i.e., COCA Academic-based bigram range, COCA Academic-based trigram frequency, and percentage of core AFL units) showed a U-shaped development. In other words, the literature reviews written on the third topic (“Online learning: Pros and cons”) had lower values in these three features. Compared with the other two topics (i.e., legalization of marijuana and pacifism), online learning is a relatively familiar topic to the L2 graduate students, especially for those who have experienced online learning during the COVID pandemic. This may be the reason that the participants did not use the bigrams of higher range value (that is, the bigrams used more widely in COCA's academic register) or trigrams of higher frequency value (that is, the trigrams appearing more frequently). Likewise, the L2 graduate students may have incorporated their personal experiences in this literature review, which may reduce the need to use the units from AFL (academic Formulas List). Interestingly, the literature reviews on Topic 3 had a higher percentage of words from AWL. This may be accounted for by the fact that the topic itself is academic and common academic vocabulary is expected.

Overall, three lexical features (i.e., content word frequency, bigram range, academic lemma TTR) showed higher values in the last writing task, suggesting some improvement in the use of academic vocabulary at the end of the tutorial series. While these features are different from the ones as strong predictors of TOEFL writing performance in Kyle and Crossley (2015) , the features share some basic characteristics, such as range and frequency indices with different reference corpora (COCA Academic in this study vs. BNC written in Kyle and Crossley (2015) . The increase in academic lemma TTR is roughly in line with the findings in Mazgutova and Kormos (2015) in which TTR-based measures improved at the end of a 4-week writing program.

As for the cohesion features, the retained distinguishing ones (i.e., content word overlaps in adjacent two sentences, word2vec-based similarity across two sentences, and use of basic connectives) in this study highlight the notable changes in local cohesion across the three writing tasks. These features are not necessarily specific to the genre of literature reviews, though, or reflect improvement in academic writing made by the L2 graduate students. Compared to the findings in the studies that used the same or similar measures, these three cohesion features have not been reported as major predictors of writing performance yet. In Crossley et al. (2016) study of descriptive essays written by L2 university students in EAP courses, they identified 32 cohesion features that were significantly correlated to students' essay scores. Nevertheless, some of the relevant cohesion features were found to be significantly correlated with essay scores in Crossley et al. (2016) , for example, adjacent overlaps of words in general across two sentences ( r = 0.32), LSA-based similarity from initial to middle paragraphs ( r = 0.23), and use of conjunctions ( r = 0.24). Therefore, we may speculate that the increase in these cohesive features may be related to the content of the online tutorials, specifically the ones about academic vocabulary and sentence structures, thus contributing positively to writing performance.

The syntactic features also showed some unexpected patterns as all four features ended up with smaller values in the last writing task. This suggests that L2 graduate students' writing became less complex syntactically at both phrasal and clausal levels, likely as a result of addressing raters' comments on long and complex sentences found in previous writing tasks. For example, the last literature reviews appeared to have fewer dependents per nominal structure, fewer clausal complements, coordinate phrases, and complex nominals per clause. Since previous studies have established positive correlations between syntactic complexity features and writing performance ( Kyle, 2016 ), we may suppose that with a relatively abstract topic (i.e., pacifism), the last literature review task may have presented some challenges to L2 graduate students who may be less able to manage syntactic structures at the expense of other writing aspects such as topic familiarity. While it is difficult to be certain about the causes of the decreases, it is equally important to keep in mind that the majority of the syntactic features (21 out of 25) examined did not have statistically significant changes across the three writing tasks. It appears that the L2 graduate students have reached a relatively stable level of syntactic development and only a handful of features fluctuated in their writing samples.

5.2. Citation practices

In terms of the frequency of citations, the literature reviews written by the L2 graduate students have fewer normalized counts of citation instances or lower citation density, compared to the citation counts in the Introduction sections of chemical engineering papers reported in Mansourizadeh and Ahmad (2011) . Meanwhile, similar to the dominant use of non-integral citations found in both expert papers and novice papers in their study, the writing samples in this study also showed a strong preference for non-integral citations (85.1 to 89.1% of raw citation counts). However, it is worth mentioning that disciplinary differences in preferred citation forms exist ( Hu and Wang, 2014 ). For example, Hu and Wang noted that published research articles in applied linguistics tend to have a more balanced distribution of integral and non-integral citations, compared to the articles in general medicine. Lillis et al. (2010) reported higher proportions of non-integral citations (over 70%) in the research articles in psychology. Considering that the topics in our study have an orientation toward social science or humanities, we may expect to see a somewhat balanced use of both citation forms as well. It requires further inquiry to understand why the L2 graduate students in this study chose to use more non-integral citations as several factors may affect a writer's preference for citation forms. In a quantitative analysis of the relationships among individual factors and citation competence, Ma and Qin (2017) reported that cognitive proficiency in source use (operationalized as knowledge of source use and ability to detect plagiarism) can significantly influence English major students' citation competence as measured with intertextual strategies, writers' stance, citation typology, and citation function in a read-to-write task. Meanwhile, students' academic reading proficiency exerted a direct impact on citation competence.

Regarding the use of reporting verbs, hedges, and boosters in the citation instances, our study shows a steady, but not statistically significant, increase in these stance-making features. More importantly, the TTRs of these lexical items increased as well, showing that the L2 graduate students may have become more aware of stance-making with these items at the end of the tutorial series. As discussed above, this progress was also related to the student's development of academic vocabulary, likely from the benefit of the tutorials on genre requirements and sentence structures. Since the changes in these features were not statistically significant, these phenomena should be interpreted with caution.

6. Implications and future studies

Before discussing the pedagogical implications of the findings, we need to point out the limitations of this study. Firstly, this study focuses on the written output of the online tutorial series. More qualitative input from the L2 graduate students as well as the instructors (e.g., feedback and scores) would supplement the quantitative analysis and facilitate the understanding of students' development of academic writing. In addition, the tutorial series has a fixed number of topics for the participants. Even with the provided bibliographies, topic familiarity may still influence the writer's performance and writing motivation. To avoid this potential issue, future studies may consider collecting writing samples from graduate students' work either from course assignments or their research projects. Lastly, for this purely online tutorial series open to voluntary participation during the pandemic, it was challenging to recruit more participants and several participants were not able to complete the whole series. Consequently, with a small sample size, the diversity in participants' background and educational level may impact the generalizability of the findings. Future projects with similar materials design may consider a hybrid mode with in-person consultation opportunities to better attract and support participants.

The format and content of the online tutorial may be useful for EAP practitioners and researchers. The materials in the tutorial series were prepared with specific aspects of literature review writing in mind, covering both genre features (e.g., logic and structure) and linguistic features (e.g., academic vocabulary, sentence structure, and reported speech). These materials can be used to supplement EAP teaching. In addition, the interactive e-book and accompanying activities used in this series are made to be open-access resources. Other researchers and EAP instructors may use some of the content and activities directly from this tutorial series, with or without modifications. At the same time, the quantified linguistic features generated by the software provided rich information regarding graduate students' writing development as well as tutorial performance. These analytical tools are free and user-friendly. With basic training, practitioners may be able to gain more insight into graduate students' writing. Furthermore, the decrease in some syntactic features, along with the lack of changes in many other linguistic features, deserves some attention. These trends may be a result of missing or under-represented components in the tutorial series. For example, discipline-specific activities on the rhetorical functions of citation and roles of citation types can be added or expanded so that L2 graduate students can better understand the expectations of citation practices in their fields. In addition, more example structures of appropriate phrasal and clausal complexity levels can be incorporated into the tutorial on “sentence structures.” For the features that did vary across the writing tasks, it would be beneficial to identify the ones that are associated positively with academic writing performance so that corresponding activities can be developed to raise students' awareness of those important features.

While the developmental patterns vary across the targeted linguistic features and citation practices, the largely positive findings of this corpus-based study (i.e., progress in cohesion and academic vocabulary) are very promising for a standalone online tutorial series that requires limited interventions from instructors.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Research Office, University of Saskatchewan. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

This project is funded by the Insight Development Grants (430-2020-00179) from Social Sciences and Humanities Research Council of Canada (SSHRC).

Acknowledgments

We would like to thank SSHRC for supporting this project.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcomm.2023.1035394/full#supplementary-material

Ansarifar, A., Shahriari, H., and Pishghadam, R. (2018). Phrasal complexity in academic writing: A comparison of abstracts written by graduate students and expert writers in applied linguistics. J. English Acad. Purp. 31, 58–71. doi: 10.1016/j.jeap.2017.12.008

CrossRef Full Text | Google Scholar

Aull, L. L., and Lancaster, Z. (2014). Linguistic markers of stance in early and advanced academic writing. Written Commun. 31, 151–183. doi: 10.1177/0741088314527055

Badenhorst, C. (2018). Citation practices of postgraduate students writing literature reviews. London Rev. Educ. 16, 121–135. doi: 10.18546/LRE.16.1.11

Biber, D., Gray, B., and Poonpon, K. (2011). Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? TESOL Quart. 45, 5–35. doi: 10.5054/tq.2011.244483

Bulté, B., and Housen, A. (2014). Conceptualizing and measuring short-term changes in L2 writing complexity. J. Second Lang. Writing 26, 42–65. doi: 10.1016/j.jslw.2014.09.005

Chen, D. T. V., Wang, Y. M., and Lee, W. C. (2016). Challenges confronting beginning researchers in conducting literature reviews. Stud. Contin. Educ. 38, 47–60. doi: 10.1080/0158037X.2015.1030335

Cheng, A. (2007). Transferring generic features and recontextualizing genre awareness: Understanding writing performance in the ESP genre-based literacy framework. English Specific Purp. 26, 287–307. doi: 10.1016/j.esp.2006.12.002

Cheng, L., and Fox, J. (2008). Towards a better understanding of academic acculturation: Second language students in Canadian universities. Canadian Mod. Lang. Rev. 65, 307–333. doi: 10.3138/cmlr.65.2.307

Crossley, S. A., Kyle, K., and Dascalu, M. (2019). The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap. Behav. Res. Methods 51, 14–27. doi: 10.3758/s13428-018-1142-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Crossley, S. A., Kyle, K., and McNamara, D. S. (2016). The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. J. Second Lang. Writ. 32, 1–16. doi: 10.1016/j.jslw.2016.01.003

Crosthwaite, P., Sanhueza, A. G., and Schweinberger, M. (2021). Training disciplinary genre awareness through blended learning: An exploration into EAP students' perceptions of online annotation of genres across disciplines. J. English Acad. Purp. 53, 101021. doi: 10.1016/j.jeap.2021.101021

Dafouz, E. (2020). Undergraduate student academic writing in English-medium higher education: Explorations through the ROAD-MAPPING lens. J. English Acad. Purp. 46, 100888. doi: 10.1016/j.jeap.2020.100888

Flowerdew, J., and Forest, R. W. (2011). “Schematic structure and lexico-grammatical realization in corpus-based genre analysis: The case of research in the PhD literature review,” in Academic Writing: At the Interface of Corpus and Discourse , eds. M. Charles, D. Pecorari, and S. Hunston (London, UK: Bloomsbury Publishing) 15–26.

Google Scholar

Hu, G., and Wang, G. (2014). Disciplinary and ethnolinguistic influences on citation in research articles. J. English Acad. Purp. 14, 14–28. doi: 10.1016/j.jeap.2013.11.001

Hyland, K. (1999). Academic attribution: Citation and the construction of disciplinary knowledge. Appl. Linguist. 20, 341–367. doi: 10.1093/applin/20.3.341

Kyle, K. (2016). Measuring syntactic development in L2 writing: Fine grained indices of syntactic complexity and usage-based indices of syntactic sophistication . Unpublished doctoral dissertation. Georgia State University.

Kyle, K., and Crossley, S. (2016). The relationship between lexical sophistication and independent and source-based writing. J. Second Lang. Writ. 34, 12–24. doi: 10.1016/j.jslw.2016.10.003

Kyle, K., and Crossley, S. (2017). Assessing syntactic sophistication in L2 writing: A usage-based approach. Lang. Test. 34, 513–535. doi: 10.1177/0265532217712554

Kyle, K., and Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quart. 49, 757–786. doi: 10.1002/tesq.194

Lea, M. R., and Street, B. V. (1998). Student writing in higher education: An academic literacies approach. Stud. Higher Educ. 23, 157–172. doi: 10.1080/03075079812331380364

Lee, J. J., Hitchcock, C., and Elliott Casal, J. (2018). Citation practices of L2 university students in first-year writing: Form, function, and stance. J. English Acad. Purp. 33, 1–11. doi: 10.1016/j.jeap.2018.01.001

Li, Y., Ma, X., Zhao, J., and Hu, J. (2020). Graduate-level research writing instruction: Two Chinese EAP teachers? localized ESP genre-based pedagogy. Jou of Eng for Acad Purp , 43. doi: 10.1016/j.jeap.2019.100813

Liardét, C. L., and Black, S. (2019). “So and so” says, states and argues: A corpus-assisted engagement analysis of reporting verbs. J. Second Lang. Writ. 44, 37–50. doi: 10.1016/j.jslw.2019.02.001

Lillis, T., Hewings, A., Vladimirou, D., and Curry, M. J. (2010). The geolinguistics of English as an academic lingua franca: citation practices across English-medium national and English-medium international journals. Int. J. Appl. Linguist. 20, 111–135. doi: 10.1111/j.1473-4192.2009.00233.x

Ma, R., and Qin, X. (2017). Individual factors influencing citation competence in L2 academic writing. J. Quantit. Linguist. 24, 213–240. doi: 10.1080/09296174.2016.1265793

Mansourizadeh, K., and Ahmad, U. K. (2011). Citation practices among non-native expert and novice scientific writers. J. English Acad. Purp. 10, 152–161. doi: 10.1016/j.jeap.2011.03.004

Marti, L., Yilmaz, S., and Bayyurt, Y. (2019). Reporting research in applied linguistics: The role of nativeness and expertise. J. English Acad. Purp. 40, 98–114. doi: 10.1016/j.jeap.2019.05.005

Mazgutova, D., and Kormos, J. (2015). Syntactic and lexical development in an intensive English academy purposes programme. J. Second Lang. Writ. 29, 3–15. doi: 10.1016/j.jslw.2015.06.004

McEnery, T., and Hardie, A. (2012). Corpus Linguistics: Method, Theory and Practice. London: Cambridge University Press. doi: 10.1017/CBO9780511981395

Okuda, T., and Anderson, T. (2018). Second language graduate students' experiences at the Writing Center: A language socialization perspective. TESOL Quart. 52, 391–413. doi: 10.1002/tesq.406

Riazi, A. M. (2016). Comparing writing performance in TOEFL-iBT and academic assignments: An exploration of textual features. Assess. Writ. 28, 15–27. doi: 10.1016/j.asw.2016.02.001

Rosmawati, R. (2014). Dynamic development of complexity and accuracy: A case study in second language academic writing. Australian Rev. Appl. Linguist. 37, 75–100. doi: 10.1075/aral.37.2.01ros

Shahsavar, Z., and Kourepaz, H. (2020). Postgraduate students' difficulties in writing their theses literature review. Cogent Educ. 7, 1784620. doi: 10.1080/2331186X.2020.1784620

Shi, L., and Dong, Y. (2015). Graduate writing assignments across faculties in a Canadian university. Canadian J. Higher Educ. 45, 123–142. doi: 10.47678/cjhe.v45i4.184723

Storch, N., and Tapper, J. (2009). The impact of an EAP course on postgraduate writing. J. English Acad. Purp. 8, 207–223. doi: 10.1016/j.jeap.2009.03.001

Swales, J., and Feak, C. (2012). Academic Writing for Graduate Students (3rd ed.) . Ann Arbor, MI: University of Michigan Press. doi: 10.3998/mpub.2173936

Terpilowski, M. (2019). scikit-posthocs: Pairwise multiple comparison tests in Python. J. Open Source Softw. 4, 1169. doi: 10.21105/joss.01169

Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., et al. (2020). SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods 17, 261–272. doi: 10.1038/s41592-019-0686-2

Xu, F. (2019). “Learning the language to write for publication: The nexus between the linguistic approach and the genre approach,” in Novice Writers and Scholarly Publication (Springer International Publishing) 117–134. doi: 10.1007/978-3-319-95333-5_7

Zhang, Z. (2011). A nested model of academic writing approaches: Chinese international graduate students' views of English academic writing. Lang. Liter. 13, 39–59. doi: 10.20360/G27G6R

Keywords: literature review writing, ESL graduate students, online tutorials, corpus-based analysis, linguistic complexity, citation practices, reporting verbs

Citation: Li Z, Makarova V and Wang Z (2023) Developing literature review writing and citation practices through an online writing tutorial series: Corpus-based evidence. Front. Commun. 8:1035394. doi: 10.3389/fcomm.2023.1035394

Received: 02 September 2022; Accepted: 10 February 2023; Published: 02 March 2023.

Reviewed by:

Copyright © 2023 Li, Makarova and Wang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Zhi Li, z.li@usask.ca

This article is part of the Research Topic

Emerging Technologies and Writing: Pedagogy and Research

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jan 4, 2024 10:52 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

CHAPTER II LITERATURE REVIEW 2.1 The Definition of Writing Skill

ABOUT WRITING SKILL

Related Papers

Journal of NELTA Gandaki

Pitamber Gautam

QuaderniCIRD

Monica Randaccio

The aim of this paper is threefold. First, a diachronic development of ESL (English as a Second Language) composition theory from the 1950s to the 1990s will be outlined. Second, the organising principles relevant to ESL composition theory will be analysed. Third, a survey of the material used in my English classes and a presentation of the introductory unit on writing skills will be given. Finally, some tentative conclusions derived from my students’ compositions will be drawn.

Damianos A . Damianopoulos

Noorlaila Hayani

Review of Research in Education

Nancy L Stein

Juan José Prat Ferrer

mary maguire

Villamor Dumlao

Writing is one of the four macro skills in English. It is where one produce concepts, ideas and thoughts through written words in a structured and well-organized manner. It is very essential to academic and professional level as it enables one to think logically and creatively. In addition, it is considered as the hardest skill to acquire. The purpose of this study is to identify the hindrances in the enhancement of English writing skills. The study utilized descriptive method of research to identify the existing barriers in enhancement of the writing skills in English medium. Significantly, descriptive research used mixed method to gather information from the participants. The descriptive study also utilized probability simple random sampling approach. Self-composed standardized questionnaires with four-point Likert scale were used where it underwent a reliability analysis and obtained a Cronbach alpha of .908 which showed that it is a reliable tool. Moreover, data are interpreted by getting the mean of every items answered by the participants from the survey questionnaire to get the descriptive value for the quantitative part. On the other hand, thematic analysis for the qualitative part to extract emerging concepts. The participants of this study were the Grade 9 students of St. Rose Catholic School, Inc. There are two hundred thirty-three (233) students from Grade 9 and a total of one hundred forty-five students (145) were randomly selected as participants as calculated in Slovein’s formula. The results showed the different hindrances in composition skills such as grammatical errors and technical errors kept learners from further improvement of their skills and these difficulties were not addressed on their lower years where they are supposed to learn the basics of writing. Hence, resulting into incompetent composition skills. Keywords: Macro Skills, English, Concepts and Ideas, Academic and Professional Level, Writing, Writing Skills, Produce Concepts

Nopi Surianto

The problem of the research is factors that influencing the students' writing skill. This research was conducted at English Education Department FKIP Lancang Kuning University, Pekanbaru. Using Descriptive Research focusing on the second year students of class A consisting of twenty five students. This research tries to explain the factors influencing the improvement of the students' writing skill. The instruments of this research were observation, field notes, and interview. After analyzing the data, it was found that factors influence the improvement of the students' writing skill were material, media, classroom activities, classroom management, teacher's strategy, and teacher's approach. In conclusion, materials, media, classroom activities, classroom management, teacher's strategy, and teacher's approach are factors that influence the students' writing skill.

Studies in Writing

Per Henning Uppstad

RELATED PAPERS

Yeni Yüzyıl Journal of Medical Sciences / Yeni Yuzyil Journal of Medical Sciences

Yeni Yüzyıl Journal of Medical Sciences - JMS

Journal of Histochemistry & Cytochemistry

fiorenzo conti

Biophysical Journal

Danielle Basore

World Development

Journal of Crystal Growth

Oscar Rodríguez Reyes

Kelly Cline

Munawar Mirza

Educational Management Administration &amp; Leadership

Javiera Marfán

The American Journal of Tropical Medicine and Hygiene

Zelalem Abebe

The Journal of Extension

John Ricketts

Raega - O Espaço Geográfico em Análise

José Oliveira

Kristina Urbanc

International Journal of Environmental Research and Public Health

ACM SIGPLAN Notices

Ivar Jacobson

Frontline Magazine

Dr.M.Aslam Parvaiz

Anticancer research

Mariangela Massaccesi

Journal of mathematics and informatics

verdiana grace masanja

Estudios de Derecho Penal Internacional

Hector Olasolo

The Spine Journal

Anuario De Derecho Penal Y Ciencias Penales

Cesar Armando Jimenez Hernandez

Indian Journal of Science and Technology

rabia shakoor

TNU Journal of Science and Technology

卡尔加里大学毕业证书办理成绩单购买 加拿大文凭办理卡尔加里大学文凭学位证书

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

• LEARNING SKILLS
• Writing a Dissertation or Thesis

Researching and Writing a Literature Review

Search SkillsYouNeed:

Learning Skills:

• A - Z List of Learning Skills
• What is Learning?
• Learning Approaches
• Learning Styles
• 8 Types of Learning Styles
• Understanding Your Preferences to Aid Learning
• Lifelong Learning
• Decisions to Make Before Applying to University
• Top Tips for Surviving Student Life
• Living Online: Education and Learning
• 8 Ways to Embrace Technology-Based Learning Approaches
• Critical Thinking Skills
• Critical Thinking and Fake News
• Understanding and Addressing Conspiracy Theories
• Critical Analysis
• Study Skills
• Exam Skills
• How to Write a Research Proposal
• Ethical Issues in Research
• Dissertation: The Introduction
• Writing your Methodology
• Dissertation: Results and Discussion
• Dissertation: Conclusions and Extras

Writing Your Dissertation or Thesis eBook

Part of the Skills You Need Guide for Students .

• Research Methods
• Teaching, Coaching, Mentoring and Counselling
• Employability Skills for Graduates

Subscribe to our FREE newsletter and start improving your life in just 5 minutes a day.

You'll get our 5 free 'One Minute Life Skills' and our weekly newsletter.

We'll never share your email address and you can unsubscribe at any time.

A literature review demonstrates that you have read around your topic and have a broad understanding of previous research, including its limitations.

In the literature review, you summarise the main viewpoints and important facts that you encountered in your reading as they relate to your chosen topic . You will also use the literature review to justify the value of doing research on your topic by showing what is already known, what is not yet known, and how it is relevant.

Your literature review should not simply be descriptive but should also provide a critical analysis of the body of work, and demonstrate that you understand how it fits together as a whole and how your own research fits with previous studies.

A key aspect of a literature review is what sources you select to include, and which you exclude.

Finding Sources

Thanks to the internet, literature searches are now relatively easy, and can be done from the comfort of your own laptop without needing to go anywhere near a library.

However, a word of warning is in order here. The ease with which anybody can access and publish to the internet means that many items published online have not been scrutinised by anybody other than the author.

In other words, content has not necessarily been checked, you have no way of knowing whether the author’s facts and claims are at all accurate and you could easily land yourself in trouble by blindly following or citing from online sources. 

Furthermore, because items on the internet are frequently changed, you may find that something you read yesterday is no longer available in the same form today. However, internet sources can be very useful for up-to-date information, especially current affairs or ongoing or very recent research.

Blogs and sites like the encyclopaedia Wikipedia are particularly prone to these problems.  For these reasons, a general rule of thumb is that you should only rely on internet resources from the websites of organisations whose information you already know to be reputable, like SkillsYouNeed.

See our page: Assessing Internet Information for more.

Do not underestimate how much physical libraries and librarians may be able to help you.

Librarians are usually hugely experienced in using all the search tools and databases, and can often show you much quicker ways of doing things, as well as tips and tricks to help you refine your search.

Furthermore, libraries may have copies of books and academic journals that are not available online. So a trip to your library may prove to be very helpful.

If you haven’t already done so, get yourself an ATHENS account through your university and/or school library. Spend time working out which of the available databases are going to be most useful for your topic, including asking the librarians for advice.

A simple way to get started with finding appropriate materials is simply to ask people who are likely to know.

You might for example ask your tutor or supervisor, or an expert or practitioner working on your chosen topic. Often, they will be able to give you some very helpful ideas about where to begin your reading.

However, be aware that some professionals may find constant requests for information intrudes on their time. Always be courteous and sensitive to the level of demand you may be making on someone’s time.

See our page: Sources of Information for more about the types of resources that you might use and how to access them.

Choosing and Refining your Search Terms

Your search terms are one of the most important elements of finding the right sources for your research project and developing them is an ongoing process.

It’s a good idea to start with a phrase that you think others will have used about the topic, perhaps that you have identified from your lectures and/or earlier study. You will probably find that your first few searches don’t turn up much that’s useful.

Use the one or two articles that you find that are on the right lines to identify alternative search terms, and continue to search until you turn up useful articles.

You can also use a tool such as Google Adword Keyword Research Tool to identify phrases and keywords that are similar to your chosen term(s). This tool is usually used by internet marketing professionals to help them find keywords similar to their own but can be useful for academic research too.

If you’re really struggling to find articles on the right topic, but you’re certain that they must be out there, drop your supervisor a note asking about possible search terms. Tell them what you’ve already used, and ask them for a few alternatives to get you started. However, this should be a last resort, as you don’t want to demonstrate your ignorance too obviously!

Finally, keep searching. You need to read a lot of sources to find the most relevant and will probably end up discarding more than half of what you read. Use abstracts to decide which articles are worth reading, and don’t read those that aren’t relevant: keep checking back to your research questions and decide whether each article is useful. If not, move on.

Critical Reading

Your literature review should not only show that you have been reading a range of materials related to your topic, but also that you have been reading them critically and have thought about the wider contexts and how they apply to your own area of research.

Critical reading is a skill that, like any other skill, is acquired with practice.

In essence, reading critically means that you do not take the claims at face value: you question the basis for claims, why the author may have done and said things in the particular way he or she did, what the wider context is, and whose interests are being served by the claims you encounter.

See our page, Critical Reading for more information.

How Many Sources?

Your university or college supervisor will be able to give you an idea of how many sources you should include in your literature review.

You will probably need to read at least double that number to find enough that are suitable for inclusion. You should also try to find several different sorts of sources: books, journal articles, dissertations, conference papers, working papers, and so on.

You need to make sure that you identify the key texts for the subject. Check a few references, and see which texts are cited most often, or ask the librarians how to use the databases to check how often each article is cited. A good way to identify when you have read enough is if your reading keeps turning up the same points and you’re not learning anything new.

A Note on Dates

There are some theories or articles which are so important in a particular field that they need to be cited, however long ago they were originally published. But those apart, you should generally prefer more recent sources published in the last five or ten years. As a rough guide, the balance of publication dates should be about two thirds from the last 10 years, and no more than one third older than that.

Writing your Literature Review

In general, your literature review should start with one or two broad paragraphs, demonstrating your understanding of the breadth of your area of study.

You should then discuss the literature that deals with your area of research and, finally, consider and critique the studies that are most directly relevant.

You should spend most time on the latter.

Writing your literature review should be an iterative process.

The best way to do it is probably to summarise each source as you go along, referencing it carefully, and grouping your sources by themes.

You will almost certainly find that the themes develop as you go along, and so do your search terms. Use headings to store your summaries and then write a more polished section under that heading when you have enough sources to be able to ‘compare and contrast’ opposing views, and particularly to draw out areas where there is disagreement and/or conflicting evidence as these are the most fruitful for further research.

Where there are gaps, you can then go back and search for more sources on that area. The best literature reviews are not only descriptive, but draw together similar thinking and provide a critical analysis of the previous research, including highlighting really good studies, or identifying flaws and gaps.

To make sure that you carrying out a critical analysis, make sure that you ask yourself the question ‘ Do I agree with this viewpoint? Why? ’, and also consider whether the methods used are strong or weak and why. This will also help you to decide on your own methodology.

Another way of checking whether you are evaluating or merely describing is to look at whether you have discussed work chronologically (likely to be descriptive) or in terms of whether there is general agreement on a topic (much more likely to be evaluative).

Checklist of Questions for Critical Reading

Ask yourself the following questions to decide whether or not a particular piece of work is worth including in your literature review.

• Who is the author? What can I find out about him/her? Has he/she written other books, articles etc.?
• What is the author’s position in the research process, e.g., gender, class, politics, life experience, relationship to research participants?
• Where and when was the document produced? What type of document is it?
• Is it reporting original research that the author has done, or is it presenting second-hand information about a topic?
• Is it formal or informal?
• Is it 'authoritative' (e.g., academic, scientific) or 'popular' (newspaper or magazine article)?
• How has it been produced? Is it glossy, with lots of pictures, diagrams, etc.?
• If it is contained on a website, is the website from a reputable organisation, or is the document drawn from some other reputable source?

The Message

• What is being said?
• What is not being said?
• How is the argument presented? Why?
• What use has been made of diagrams, pictures etc.?
• Who was or is the intended audience?
• Whose interests are being served by this message? Are there political implications, for instance?
• What evidence is presented to support the claims that are made?
• Does the evidence actually support the claims? Is the evidence presented in enough detail for you to make up your own mind whether you agree with the claims?
• Are there errors or inconsistencies?
• What is the significance to my topic and the research that I wish to carry out?

Your literature review should also demonstrate how your study does or will relate to previous work, and how it either fills gaps, or responds to calls for further work.

Your literature review will help you to refine your research question. It should also help you to explain how your methodology fits with previous work, and help you to identify and evaluate possible research methods.

A Note on Tense

When you are describing someone’s findings or opinions, it is probably best to use the past tense.

For example:

“Jones (2001) argued that…”.

Many authors of academic papers prefer the present tense when describing opinions or views (“Jones (2001) argues that…”). However, it is always possible that Jones has subsequently changed his/her view, and therefore the past tense is preferable.

The past tense is always going to be correct for something that was expressed in the past; the present tense may no longer be true.

Citations and References

Your university will almost certainly have a preferred style for citations and references that you will need to use. Make sure you understand how this works before you start writing your literature review and use it consistently throughout.

Keep your references up to date as you go, and make sure that you always cite the reference as you write: it’s much easier than trying to build a reference list at the end.

See our page on Academic Referencing for more information

For scientific subjects, Vancouver (numerical) referencing is often preferred.

However, it is much harder to check that your references are correct using this system. It is therefore better to use a (name, date) system of citations until you are certain that you have finished revising the document.

Alternatively, use a system of end-notes which will automatically update the numbering if you move a citation as you will otherwise end up hopelessly confused.

Draft, Draft and Redraft

Finally, once you have written each section by theme, go back and read the whole thing to check that the sections flow logically one from another, and that the whole literature review reads sensibly and coherently.

As with any essay or extended piece of writing, editing and redrafting will improve the quality of your writing, as will asking someone else to read it over and check for errors or inconsistencies.

You should also do a search to check for consistent use of British or American spellings (-ise and -ize, for example), double spaces after words, and double/single inverted commas around quotations. You might think such details are less important than the content, but the marker may not share your view.

Continue to: Writing a Research Proposal Writing a Dissertation: The Introduction

See also: Writing a Methodology Dissertation Results and Discussion Dissertation Conclusions and Extra Sections

Writing Narrative Literature Reviews: Build the Skills for Your Success

by Study Smart

Academic Academic Academic Literacy

Tue, Apr 16, 2024

1 PM – 2 PM (GMT+10)

Online Event

Registration

Main Navigation

• Accept offer and enrol
• Current Students

Personalise your experience

Did you mean..., diploma of arts and social sciences, art/science collaboration wins waterhouse natural science art prize, unit of study educ7002 on the shoulders of giants: critically examining the literature in education (2025).

Future students: T: 1800 626 481 E: Email your enquiry here

Current students: Contact: Faculty of Education

Students studying at an education collaboration: Please contact your relevant institution

updated - DO NOT REMOVE THIS LINE 6:05 AM on Fri, 12 April

Show me unit information for year

Unit snapshot.

PG Coursework Unit

Credit points

Faculty & college.

Faculty of Education

Pre-requisites

EDUC7001 - What is Educational Research?

Enrolment information

Attendance at the intensive workshops is compulsory for this unit. There are two x two-day on-site workshops (4 days in total) on the Gold Coast campus.

Unit description

Equips students to critically examine the literature in their agreed area for identification of research opportunities. Students are supported in their development of skills in critical reading, thinking and analysis; perspective and impact; context, purpose and the function of texts. The unit will also enable students to manage the demands of locating, synthesising/analysing and using various texts to develop competence in academic writing, in order to prepare for their thesis-based literature review. 

Unit content

Availabilities.

Not currently available in 2025

Learning outcomes

Unit Learning Outcomes express learning achievement in terms of what a student should know, understand and be able to do on completion of a unit. These outcomes are aligned with the graduate attributes . The unit learning outcomes and graduate attributes are also the basis of evaluating prior learning.

On completion of this unit, students should be able to:

locate, organise and summarise information relevant to an agreed research area

analyse, critique and synthesise information relevant to an agreed research area

formulate and write a critical literature review

demonstrate excellent text based organisational referencing and citation skills

Fee information

Commonwealth Supported courses For information regarding Student Contribution Amounts please visit the Student Contribution Amounts .

Fee paying courses For postgraduate or undergraduate full-fee paying courses please check Domestic Postgraduate Fees OR Domestic Undergraduate Fees .

International

Please check the international course and fee list to determine the relevant fees.

Any questions? We'd love to help

• Open access
• Published: 10 April 2024

Development of an index system for the scientific literacy of medical staff: a modified Delphi study in China

• Shuyu Liang 2   na1 ,
• Ziyan Zhai 2   na1 ,
• Xingmiao Feng 2 ,
• Xiaozhi Sun 1 ,
• Jingxuan Jiao 1 ,
• Yuan Gao 1   na2 &
• Kai Meng   ORCID: orcid.org/0000-0003-1467-7904 2 , 3   na2  

BMC Medical Education volume  24 , Article number:  397 ( 2024 ) Cite this article

118 Accesses

Metrics details

Scientific research activity in hospitals is important for promoting the development of clinical medicine, and the scientific literacy of medical staff plays an important role in improving the quality and competitiveness of hospital research. To date, no index system applicable to the scientific literacy of medical staff in China has been developed that can effectively evaluate and guide scientific literacy. This study aimed to establish an index system for the scientific literacy of medical staff in China and provide a reference for improving the evaluation of this system.

In this study, a preliminary indicator pool for the scientific literacy of medical staff was constructed through the nominal group technique ( n  = 16) with medical staff. Then, two rounds of Delphi expert consultation surveys ( n  = 20) were conducted with clinicians, and the indicators were screened, revised and supplemented using the boundary value method and expert opinions. Next, the hierarchical analysis method was utilized to determine the weights of the indicators and ultimately establish a scientific literacy indicator system for medical staff.

Following expert opinion, the index system for the scientific literacy of medical staff featuring 2 first-level indicators, 9 second-level indicators, and 38 third-level indicators was ultimately established, and the weights of the indicators were calculated. The two first-level indicators were research literacy and research ability, and the second-level indicators were research attitude (0.375), ability to identify problems (0.2038), basic literacy (0.1250), ability to implement projects (0.0843), research output capacity (0.0747), professional capacity (0.0735), data-processing capacity (0.0239), thesis-writing skills (0.0217), and ability to use literature (0.0181).

Conclusions

This study constructed a comprehensive scientific literacy index system that can assess medical staff's scientific literacy and serve as a reference for evaluating and improving their scientific literacy.

Peer Review reports

Due to the accelerated aging of the population and the growing global demand for healthcare in the wake of epidemics, there is an urgent need for medicine to provide greater support and protection. Medical scientific research is a critical element in promoting medical science and technological innovation, as well as improving clinical diagnosis and treatment techniques. It is the main driving force for the development of healthcare [ 1 ].

Medical personnel are highly compatible with clinical research. Due to their close interaction with patients, medical staff are better equipped to identify pertinent clinical research issues and actually implement clinical research projects [ 2 ]. Countries have created favorable conditions for the research and development of medical personnel by providing financial support, developing policies, and offering training courses [ 3 , 4 ]. However, some clinical studies have shown that the ability of most medical staff does not match current health needs and cannot meet the challenges posed by the twenty-first century [ 5 ]. It is clear that highly skilled professionals with scientific literacy are essential for national and social development [ 6 ]. Given the importance of scientific research in countries and hospitals, it is crucial to determine the level of scientific research literacy that medical personnel should possess and how to train them to acquire the necessary scientific research skills. These issues have significant practical implications.

Scientific literacy refers to an individual's ability to engage in science-related activities [ 7 ]. Some scholars suggest that the scientific literacy of medical personnel encompasses the fundamental qualities required for scientific research work, encompassing three facets: academic moral accomplishment, scientific research theory accomplishment, and scientific research ability accomplishment [ 8 ]. The existing research has focused primarily on the research capabilities of medical staff. According to Rillero, problem-solving skills, critical thinking, communication skills, and the ability to interpret data are the four core components of scientific literacy [ 9 ]. The ability to perform scientific research in nursing encompasses a range of abilities, including identifying problems, conducting literature reviews, designing and conducting scientific research, practicing scientific research, processing data, and writing papers [ 10 ]. Moule and Goodman proposed a framework of skills that research-literate nurses should possess, such as critical thinking capacity, analytical skills, searching skills, research critique skills, the ability to read and critically appraise research, and an awareness of ethical issues [ 11 ]. Several researchers have developed self-evaluation questionnaires to assess young researchers' scientific research and innovative abilities in the context of university-affiliated hospitals (UHAs) [ 12 ]. The relevant indicators include sensitivity to problems, sensitivity to cutting-edge knowledge, critical thinking, and other aspects. While these indicators cover many factors, they do not consider the issue of scientific research integrity in the medical field. The lack of detailed and targeted indicators, such as clinical resource collection ability and interdisciplinary cooperation ability, hinders the effective measurement of the current status of scientific literacy among medical staff [ 12 ]. In conclusion, the current research on the evaluation indicators of scientific literacy among medical personnel is incomplete, overlooking crucial humanistic characteristics, attitudes, and other moral literacy factors. Therefore, there is an urgent need to establish a comprehensive and systematic evaluation index to effectively assess the scientific literacy of medical staff.

Therefore, this study utilized a literature search and nominal group technique to screen the initial evaluation index and subsequently constructed an evaluation index system for medical staff's scientific research literacy utilizing the Delphi method. This index system would serve as a valuable tool for hospital managers, aiding them in the selection, evaluation, and training of scientific research talent. Additionally, this approach would enable medical personnel to identify their own areas of weakness and implement targeted improvement strategies.

Patient and public involvement

Patients and the public were not involved in this research.

Study design and participants

In this study, an initial evaluation index system was developed through a literature review and nominal group technique. Subsequently, a more comprehensive and scientific index system was constructed by combining qualitative and quantitative analysis utilizing the Delphi method to consult with experts. Finally, the hierarchical analysis method and the percentage weight method were employed to empower the index system.

The program used for this study is shown in Fig.  1 .

Study design. AHP, analytic hierarchy process

Establishing the preliminary indicator pool

Search process.

A literature search was performed in the China National Knowledge Infrastructure (CNKI), WanFang, PubMed, Web of Science and Scopus databases to collect the initial evaluation indicators. The time span ranged from the establishment of the database to July 2022. We used a combination of several MeSH terms in our searches:(("Medical Staff"[Mesh] OR "Nurses"[Mesh] OR "Physicians"[Mesh])) AND (("Literacy"[Mesh]) OR "Aptitude"[Mesh]). We also used several Title/Abstract searches, including keywords such as: Evaluation, scientific literacy, research ability.

The inclusion criteria were as follows: (1)The subjects were nurses, medicial staff and other personnel engaged in the medical industry; (2) Explore topics related to scientific literacy, such as research ability, and literature that can clarify the structure or dependency between indicators of scientific literacy; (3) Select articles published in countries such as China, the United States, the United Kingdom, Australia and Canada; (4) Research published in English or Chinese is considered to be eligible. The exclusion criteria are as follows: (1) indicators not applicable to medical staff; (2) Conference abstracts, case reports or review papers; (3) Articles with repeated descriptions; (4) There are no full-text articles or grey literature. A total of 78 articles were retrieved and 60 were retained after screening according to inclusion and exclusion criteria.

The research was conducted by two graduate students and two undergraduate students who participated in the literature search and screening. The entire research process was supervised and guided by one professor. All five members were from the fields of social medicine and health management. The professor was engaged in hospital management and health policy research for many years.

Nominal group technique

The nominal group technique was introduced at Hospital H in Beijing in July 2022. This hospital, with over 2,500 beds and 3,000 doctors, is a leading comprehensive medical center also known for its educational and research achievements, including numerous national research projects and awards.

The interview questions were based on the research question: What research literacy should medical staff have? 16 clinicians and nurses from Hospital H were divided into 2 equal groups and asked to provide their opinions on important aspects of research literacy based on their positions and experiences. Once all participants had shared their thoughts, similar responses were merged and polished. If anyone had further inputs after this, a second round of interviews was held until no new inputs were given. The entire meeting, including both rounds, was documented by researchers with audio recordings on a tape recorder.

Scientific literacy dimensions

Based on the search process, the research group extracted 58 tertiary indicators. To ensure the practicality and comprehensiveness of the indicators, the Nominal group technique was used on the basis of the literature search. Panelists summarized the entries shown in the interviews and merged similar content to obtain 32 third-level indicators. The indicators obtained from the literature search were compared. Several indicators with similar meanings, such as capture information ability, language expression ability, communication ability, and scientific research integrity, were merged. Additionally, the indicators obtained from the literature search, such as scientific research ethics, database use ability, feasibility and analysis ability, were added to the 15 indicators. A total of 47 third-level indicators were identified.

Fengling Dai and colleagues developed an innovation ability index system with six dimensions covering problem discovery, information retrieval, research design, practice, data analysis, and report writing, which represents the whole of innovative activity. Additionally, the system includes an innovation spirit index focusing on motivation, thinking, emotion, and will, reflecting the core of the innovation process in terms of competence [ 13 ]. Liao et al. evaluated the following five dimensions in their study on scientific research competence: literature processing, experimental manipulation, statistical analysis, manuscript production, and innovative project design [ 14 ]. Mohan claimed that scientific literacy consists of four core components: problem solving, critical thinking, communication skills, and the ability to interpret data [ 15 ].

This study structured scientific literacy into 2 primary indicators (research literacy and research competence) and 9 secondary indicators (basic qualifications, research ethics, research attitude, problem identification, literature use, professional capacity, subject implementation, data processing, thesis writing, and research output).

Using the Delphi method to develop an index system

Expert selection.

This study used the Delphi method to distribute expert consultation questionnaires online, allowing experts to exchange opinions anonymously to ensure that the findings were more desirable and scientific. No fixed number of experts is required for a Delphi study, but the more experts involved, the more stable the results will be [ 16 ]; this method generally includes 15 to 50 experts [ 17 ]. We selected clinicians from several tertiary hospitals in the Beijing area to serve as Delphi study consultants based on the following inclusion criteria: (1) they had a title of senior associate or above; (2) they had more than 10 years of work experience in the field of clinical scientific research, and (3) they were presiding over national scientific research projects. The exclusion criteria were as follows: (1) full-time scientific researchers, and (2) personnel in hospitals who were engaged only in management. To ensure that the selected experts were representative, this study selected 20 experts from 14 tertiary hospitals affiliated with Capital Medical University, Peking University, the Chinese Academy of Medical Sciences and the China Academy of Traditional Chinese Medicine according to the inclusion criteria; the hospitals featured an average of 1,231 beds each, and 9 hospitals were included among the 77 hospitals in the domestic comprehensive hospital ranking (Fudan Hospital Management Institute ranking). The experts represented various specialties and roles from different hospitals, including cardiology, neurosurgery, neurology, ear and throat surgery, head and neck surgery, radiology, imaging, infection, vascular interventional oncology, pediatrics, general practice, hematology, stomatology, nephrology, urology, and other related fields. This diverse group included physicians, nurses, managers, and vice presidents. The selected experts had extensive clinical experience, achieved numerous scientific research accomplishments and possessed profound knowledge and experience in clinical scientific research. This ensured the reliability of the consultation outcomes.

Design of the expert consultation questionnaire

The Delphi survey for experts included sections on their background, familiarity with the indicator system, system evaluation, and opinions. Experts rated indicators on importance, feasibility, and sensitivity using a 1–10 scale and their own familiarity with the indicators on a 1–5 scale. They also scored their judgment basis and impact on a 1–3 scale, considering theoretical analysis, work experience, peer understanding, and intuition. Two rounds of Delphi surveys were carried out via email with 20 experts to evaluate and suggest changes to the indicators. Statistical coefficients were calculated to validate the Delphi process. Feedback from the first round led to modifications and the inclusion of an AHP questionnaire for the second round. After the second round, indicators deemed less important were removed, and expert discussion finalized the indicator weights based on their relative importance scores. This resulted in the development of an index system for medical staff scientific literacy. The questionnaire is included in Additional file 1 (first round) and Additional file 2 (second round).

Using the boundary value method to screen the indicators

In this study, the boundary value method was utilized to screen the indicators of medical staff's scientific literacy, and the importance, feasibility, and sensitivity of each indicator were measured using the frequency of perfect scores, the arithmetic mean, and the coefficient of variation, respectively. When calculating the frequency of perfect scores and arithmetic means, the boundary value was set as "mean-SD," and indicators with scores higher than this value were retained. When calculating the coefficient of variation, the cutoff value was set to "mean + SD," and indicators with values below this threshold were retained.

The principles of indicator screening are as follows:

To evaluate the importance of the indicators, if none of the boundary values of the three statistics met the requirements, the indicators were deleted.

If an indicator has two aspects, importance, feasibility, or sensitivity, and each aspect has two or more boundary values that do not meet the requirements, then the indicator is deleted.

If all three boundary values for an indicator meet the requirements, the research group discusses the modification feedback from the experts and determines whether the indicator should be used.

The results of the two rounds of boundary values are shown in Table  1 .

Using the AHP to assign weights

After the second round of Delphi expert consultations, the analytic hierarchy process (AHP) was used to determine the weights of the two first-level indicators and the nine second-level indicators. The weights of the 37 third-level indicators were subsequently calculated via the percentage weight method. The AHP, developed by Saaty in the 1980s, is used to determine the priority and importance of elements constituting the decision-making hierarchy. It is based on multicriteria decision-making (MCDM) and determines the importance of decision-makers' judgments based on weights derived from pairwise comparisons between elements. In the AHP, pairwise comparisons are based on a comparative evaluation in which each element's weight in the lower tier is compared with that of other lower elements based on the element in the upper tier [ 18 ].

AHP analysis involves the following steps:

Step 1: Establish a final goal and list related elements to construct a hierarchy based on interrelated criteria.

Step 2: Perform a pairwise comparison for each layer to compare the weights of each element. Using a score from 1 to 9, which is the basic scale of the AHP, each pair is compared according to the expert’s judgment, and the importance is judged [ 19 , 20 ].

Yaahp software was employed to analyze data by creating a judgment matrix based on the experts' scores and hierarchical model. The index system weights were obtained by combining the experts' scores. The percentage weight method used experts' importance ratings from the second round to calculate weights, ranking indicators by importance, calculating their scores based on frequency of ranking, and determining weighting coefficients by dividing these scores by the total of all third-level indicators' scores. The third-level indicator weighting coefficients were then calculated by multiplying the coefficients [ 21 ].

Data analysis

Expert positivity coefficient.

The expert positivity coefficient is indicated by the effective recovery rate of the expert consultation questionnaire, which represents the level of expert positivity toward this consultation and determines the credibility and scientific validity of the questionnaire results. Generally, a questionnaire with an effective recovery rate of 70% is considered very good [ 22 ].

In this study, 20 questionnaires were distributed in both rounds of Delphi expert counseling, and all 20 were effectively recovered, resulting in a 100% effective recovery rate. Consequently, the experts provided positive feedback on the Delphi counseling.

Expert authority coefficient (CR)

The expert authority coefficient (Cr) is the arithmetic mean of the judgment coefficient (Ca) and the familiarity coefficient (Cs), namely, Cr =  \(\frac{({\text{Ca}}+{\text{Cs}})}{2}\) . The higher the degree of expert authority is, the greater the predictive accuracy of the indicator. A Cr ≥ 0.70 was considered to indicate an acceptable level of confidence [ 23 ]. Ca represents the basis on which the expert makes a judgment about the scenario in question, while Cs represents the expert's familiarity with the relevant problem [ 24 ].

Ca is calculated on the basis of experts' judgments of each indicator and the magnitude of its influence. In this study, experts used "practical experience (0.4), "theoretical analysis (0.3), "domestic and foreign peers (0.2)" and "intuition (0.1)" as the basis for judgment and assigned points according to the influence of each basis for judgment on the experts' judgment. Ca = 1 when the basis for judgment has a large influence on the experts, and Ca = 0.5 when the influence of the experts' judgment is at a medium level. When no influence on expert judgment was evident, Ca = 0 [ 25 ] (Table  2 ).

Cs refers to the degree to which the expert was familiar with the question. This study used the Likert scale method to score experts’ familiarity with the question on a scale ranging from 0 to 1 (1 = very familiar, 0.75 = more familiar, 0.5 = moderately familiar, 0.25 = less familiar, 0 = unfamiliar). The familiarity coefficient for each expert (the average familiarity for each indicator) was calculated. The average familiarity coefficient was subsequently computed [ 26 ].

The Cr value of the primary indicator in this study was 0.83, and the Cr value of the secondary indicator was 0.82 (> 0.7); hence, the results of the expert consultation were credible and accurate, as shown in Table  3 .

The degree of expert coordination is an important indicator used to judge the consistency among various experts regarding indicator scores. This study used the Kendall W coordination coefficient test to determine the degree of expert coordination. A higher Kendall W coefficient indicates a greater degree of expert coordination and greater consistency in expert opinion, and P  <  0.05 indicates that the difference is significant [ 26 ]. The results of the three-dimensional harmonization coefficient test for each indicator in the two rounds of the expert consultation questionnaire were valid ( p  <  0.01 ), emphasizing the consistency of the experts' scores. The values of the Kendall W coordination coefficients for both rounds are shown in Table  4 .

Basic information regarding the participants

The 20 Delphi experts who participated in this study were predominantly male (80.0%) rather than female (20.0%). In addition, the participants’ ages were mainly concentrated in the range of 41–50 years old (60.0%). The majority of the experts were doctors by profession (85.0%), and their education and titles were mainly doctoral degree (90.0%) and full senior level (17.0%). The experts also exhibited high academic achievement in their respective fields and had many years of working experience, with the majority having between 21 and 25 years of experience (40.0%) (Table  5 ).

Index screening

The boundary value method was applied to eliminate indicators, leading to the removal of 6 third-level indicators in the first round. One of these, the ability to use statistical software, was associated with a more significant second-level indicator involving data processing, which was kept after expert review. Six indicators were merged into three indicators due to duplication, and 5 third-level indicators were added, resulting in 2 primary indicators, 10 secondary indicators, and 43 third-level indicators.

In the second round of Delphi expert consultation, 5 third-level indicators were deleted, as shown in Additional file 3 , and only one third-level indicator, "Scientific spirit", remained under the secondary indicator "research attitude". The secondary indicator "Research attitude" was combined with "Research ethics" and the third-level indicator "Scientific spirit" was also considered part of "Research ethics". After expert discussion, these were merged into a new secondary indicator "Research attitude" with three third-level indicators: "Research ethics", "Research integrity", and "Scientific spirit". The final index system included two primary indicators, nine secondary indicators, and thirty-eight third-level indicators, as shown in Additional File 3 .

Final index system with weights

The weights of the two primary indexes, research literacy and research ability, were equal. This was determined using the hierarchical analysis method and the percentage weight method based on the results of the second round of Delphi expert consultation (Table  6 ). The primary indicator of research literacy encompasses the fundamental qualities and attitudes medical staff develop over time, including basic qualifications and approach to research. The primary indicator of research ability refers to medical professionals' capacity to conduct scientific research in new areas using suitable methods, as well as their skills needed for successful research using scientific methods.

In this study, the Delphi method was employed, and after two rounds of expert consultation, in accordance with the characteristics and scientific research requirements of medical staff in China, an index system for the scientific literacy of medical staff in China was constructed. The index system for medical staff's scientific literacy in this study consists of 2 first-level indicators, 9 second-level indicators, and 38 third-level indicators. Medical institutions at all levels can use this index system to scientifically assess medical staff's scientific literacy.

In 2014, the Joint Task Force for Clinical Trial Competency (JTF) published its Core Competency Framework [ 27 ]. The Framework focuses more on the capacity to conduct clinical research. These include principles such as clinical research and quality practices for drug clinical trials. However, this framework does not apply to the current evaluation of scientific literacy in hospitals. Because these indicators do not apply to all staff members, there is a lack of practical scientific research, such as information about the final paper output. Therefore, the experts who constructed the index system in this study came from different specialties, and the indicators can be better applied to scientific researchers in all fields. This approach not only addresses clinical researchers but also addresses the concerns of hospital managers, and the indicators are more applicable.

The weighted analysis showed that the primary indicators "research literacy" and "research ability" had the same weight (0.50) and were two important components of scientific literacy. Research ability is a direct reflection of scientific literacy and includes the ability to identify problems, the ability to use literature, professional capacity, subject implementation capacity, data-processing capacity, thesis-writing skills, and research output capacity. Only by mastering these skills can medical staff carry out scientific research activities more efficiently and smoothly. The ability to identify problems refers to the ability of medical staff to obtain insights into the frontiers of their discipline and to identify and ask insightful questions. Ratten claimed that only with keen insight and sufficient sensitivity to major scientific issues can we exploit the opportunities for innovation that may lead to breakthroughs [ 28 ]. Therefore, it is suggested that in the process of cultivating the scientific literacy of medical staff, the ability to identify problems, including divergent thinking, innovative sensitivity, and the ability to produce various solutions, should be improved. Furthermore, this study included three subentries of the secondary indicator "research attitude", namely, research ethics, research integrity, and scientific spirit. This is likely because improper scientific research behavior is still prevalent. A study conducted in the United States and Europe showed that the rate of scientific research misconduct was 2% [ 13 ]. A small survey conducted in Indian medical schools and hospitals revealed that 57% of the respondents knew that someone had modified or fabricated data for publication [ 28 ]. The weight of this index ranked first in the secondary indicators, indicating that scientific attitude is an important condition for improving research quality, relevance, and reliability. Countries and hospitals should develop, implement, and optimize policies and disciplinary measures to combat academic misconduct.

In addition, the third-level indicator "scheduling ability" under the second-level indicator "basic qualification" has a high weight, indicating that medical staff attach importance to management and distribution ability in the context of scientific research. Currently, hospitals face several problems, such as a shortage of medical personnel, excessive workload, and an increase in the number of management-related documents [ 29 , 30 ]. These factors result in time conflicts between daily responsibilities and scientific research tasks, thereby presenting significant obstacles to the allocation of sufficient time for scientific inquiry [ 31 ]. Effectively arranging clinical work and scientific research time is crucial to improving the overall efficiency of scientific research. In the earlier expert interviews, most medical staff believed that scientific research work must be combined with clinical work rather than focused only on scientific research. Having the ability to make overall arrangements is essential to solving these problems. The high weight given to the second-level index of 'subject implementation capacity', along with its associated third-level indicators, highlights the challenges faced by young medical staff in obtaining research subjects. Before implementing a project, researchers must thoroughly investigate, analyze, and compare various aspects of the research project, including its technical, economic, and engineering aspects. Moreover, potential financial and economic benefits, as well as social impacts, need to be predicted to determine the feasibility of the project and develop a research plan [ 32 ]. However, for most young medical staff in medical institutions, executing such a project can be challenging due to their limited scientific research experience [ 33 ]. A researcher who possesses these skills can truly carry out independent scientific research.

The weights of the second-level index "research output capacity" cannot be ignored. In Chinese hospitals, the ability to produce scientific research output plays a certain role in employees’ ability to obtain rewards such as high pay, and this ability is also used as a reference for performance appraisals [ 34 ]. The general scientific research performance evaluation includes the number of projects, scientific papers and monographs, scientific and technological achievements, and patents. In particular, the publication of papers is viewed as an indispensable aspect of performance appraisal by Chinese hospitals [ 35 ]. Specifically, scientific research papers are the carriers of scientific research achievements and academic research and thus constitute an important symbol of the level of medical development exhibited by medical research institutions; they are thus used as recognized and important indicators of scientific research output [ 36 ]. This situation is consistent with the weight evaluation results revealed by this study.

The results of this study are important for the training and management of the scientific research ability of medical personnel. First, the index system focuses not only on external characteristics such as scientific knowledge and skills but also on internal characteristics such as individual traits, motivation, and attitudes. Therefore, when building a research team and selecting and employing researchers, hospital managers can use the index system to comprehensively and systematically evaluate the situation of researchers, which is helpful for optimizing the allocation of a research team, learning from each other's strengths, and strengthening the strength of the research team. Second, this study integrates the content of existing research to obtain useful information through in-depth interviews with medical staff and constructs an evaluation index system based on Delphi expert consultation science, which comprehensively includes the evaluation of the whole process of scientific research activities. These findings can provide a basis for medical institutions to formulate scientific research training programs, help medical personnel master and improve scientific research knowledge and skills, and improve their working ability and quality. Moreover, the effectiveness of the training can also be evaluated according to the system.

In China, with the emergence of STEM rankings, hospitals pay more and more attention to the scientific research performance of medical personnel. Scientific literacy not only covers the abilities of medical personnel engaged in scientific research, but also reflects their professional quality in this field. Having high quality medical personnel often means that they have excellent scientific research ability, and their scientific research performance will naturally rise. In view of this,,medical institutions can define the meaning of third-level indicators and create Likert scales to survey medical staff. Based on the weights assigned to each indicator, comprehensive scores can be calculated to evaluate the level of scientific literacy among medical staff. Through detailed data analysis, they can not only reveal their shortcomings in scientific research ability and quality, but also provide a strong basis for subsequent training and promotion. Through targeted inspection, we can not only promote the comprehensive improvement of the ability of medical staff, but also promote the steady improvement of their scientific research performance, and inject new vitality into the scientific research cause of hospitals.

Limitations

This study has several limitations that need to be considered. First, the participants were only recruited from Beijing (a city in China), potentially lacking geographical diversity. We plan to select more outstanding experts from across the country to participate. Second, the index system may be more suitable for countries with medical systems similar to those of China. When applying this system in other countries, some modifications may be necessary based on the local context. Last, While this study has employed scientific methods to establish the indicator system, the index system has yet to be implemented on a large sample of medical staff. Therefore, the reliability and validity of the index system must be confirmed through further research. In conclusion, it is crucial to conduct further detailed exploration of the effectiveness and practical application of the index system in the future.

This study developed an evaluation index system using the Delphi method to assess the scientific literacy of medical staff in China. The system comprises two primary indicators, nine secondary indicators, and thirty-eight third-level indicators, with each index assigned a specific weight. The index system emphasizes the importance of both attitudes and abilities in the scientific research process for medical staff and incorporates more comprehensive evaluation indicators. In the current era of medical innovation, enhancing the scientific literacy of medical staff is crucial for enhancing the competitiveness of individuals, hospitals, and overall medical services in society. This evaluation index system is universally applicable and beneficial for countries with healthcare systems similar to those of China. This study can serve as a valuable reference for cultivating highly qualified and capable research personnel and enhancing the competitiveness of medical research.

Availability of data and materials

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Coloma J, Harris E. From construction workers to architects: developing scientific research capacity in low-income countries. PLoS Biol. 2009;7(7):e1000156. https://doi.org/10.1371/journal.pbio.1000156 .

Article   Google Scholar  

Brauer SG, Haines TP, Bew PG. Fostering clinician-led research. Aust J Physiother. 2007;53(3):143–4. https://doi.org/10.1016/s0004-9514(07)70020-x .

The L. China’s research renaissance. Lancet. 2019;393(10179):1385. https://doi.org/10.1016/S0140-6736(19)30797-4 .

Hannay DR. Evaluation of a primary care research network in rural Scotland. Prim Health Care ResDevelop. 2006;7(3):194–200. https://doi.org/10.1191/1463423606pc296oa .

Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:1923–58.

Xie Y, Wang J, Li S, Zheng Y. Research on the Influence Path of Metacognitive Reading Strategies on Scientific Literacy. J Intell. 2023;11(5):78. https://doi.org/10.3390/jintelligence11050078 . PMID: 37233327; PMCID: PMC10218841.

Pang YH, Cheng JL. Revise of scientific research ability self-evaluation rating scales of nursing staff. Chin Nurs Res. 2011;13:1205–8. https://doi.org/10.3969/j.issn.1009-6493.2011.13.040 .

Zhang J, Jianshan MAO, Gu Y. On the cultivation of scientific research literacy of medical graduate students. Continu Med Educ China. 2023;15(3):179–82. https://doi.org/10.3969/j.issn.1674-9308.2023.03.043 .

Rillero P. Process skills and content knowledge. Sci Act. 1998;3:3–4.

Google Scholar  

Liu RS. Study on reliability and validity of self rating scale for scientific research ability of nursing staff. Chinese J Pract Nurs. 2004;9:8–10. https://doi.org/10.3760/cma.j.issn.1672-7088.2004.09.005 .

Moule P, Goodman M. Nursing research: An introduction. London, UK: Sage; 2013.

Xue J, Chen X, Zhang Z, et al. Survey on status quo and development needs of research and innovation capabilities of young researchers at university-affiliated hospitals in China: a cross-sectional survey. Ann Transl Med. 2022;10(18):964. https://doi.org/10.21037/atm-22-3692 .

Fanelli D, Costas R, Fang FC, et al. Testing hypotheses on risk factors for scientific misconduct via matched-control analysis of papers containing problematic image duplications. Sci Eng Ethics. 2019;25(3):771–89. https://doi.org/10.1007/s11948-018-0023-7 .

Liao Y, Zhou H, Wang F, et al. The Impact of Undergraduate Tutor System in Chinese 8-Year Medical Students in Scientific Research. Front Med (Lausanne). 2022;9:854132. https://doi.org/10.3389/fmed.2022.854132 .

Mohan L, Singh Y, Kathrotia R, et al. Scientific literacy and the medical student: A cross-sectional study. Natl Med J India. 2020;33(1):35–7. https://doi.org/10.4103/0970-258X.308242 .

Jorm AF. Using the Delphi expert consensus method in mental health research. Aust N Z J Psychiatry. 2015;49(10):887–97. https://doi.org/10.1177/0004867415600891 .

Xinran S, Heping W, Yule H, et al. Defining the scope and weights of services of a family doctor service project for the functional community using Delphi technique and analytic hierarchy process. Chinese Gen Pract. 2021;24(34):4386–91.

Park S, Kim HK, Lee M. An analytic hierarchy process analysis for reinforcing doctor-patient communication. BMC Prim Care. 2023;24(1):24. https://doi.org/10.1186/s12875-023-01972-3 . Published 2023 Jan 21.

Zhou MLY, Yin H, et al. New screening tool for neonatal nutritional risk in China: a validation study. BMJ Open. 2021;11(4):e042467. https://doi.org/10.1136/bmjopen-2020-042467 .

Wang K, Wang Z, Deng J, et al. Study on the evaluation of emergency management capacity of resilient communities by the AHP-TOPSIS method. Int J Environ Res Public Health. 2022;19(23):16201. https://doi.org/10.3390/ijerph192316201 .

Yuwei Z, Chuanhui Y, Junlong Z, et al. Application of analytic Hierarchy Process and percentage weight method to determine the weight of traditional Chinese medicine appropriate technology evaluation index system. Chin J Tradit Chinese Med. 2017;32(07):3054–6.

Babbie E. The practice of social research. 10th Chinese language edition. Huaxia Publisher, 2005: 253–4.

Liu W, Hu M, Chen W. Identifying the Service Capability of Long-Term Care Facilities in China: an e-Delphi study. Front Public Health. 2022;10:884514. https://doi.org/10.3389/fpubh.2022.884514 .

Zeng G. Modern epidemiological methods and application. Pecking Union Medical College Union Press, 1996.

Geng Y, Zhao L, Wang Y, et al. Competency model for dentists in China: Results of a Delphi study. PLoS One. 2018;13(3):e0194411. https://doi.org/10.1371/journal.pone.0194411 .

Cong C, Liu Y, Wang R. Kendall coordination coefficient W test and its SPSS implementation. Journal of Taishan Medical College. 2010;31(7):487–490. https://doi.org/10.3969/j.issn.1004-7115.2010.07.002 .

Sonstein S, Seltzer J, Li R, et al. Moving from compliance to competency: a harmonized core competency framework for the clinical research professional. Clin Res. 2014;28(3):17–23.

Madan C, Kruger E, Tennant M. 30 Years of dental research in Australia and India: a comparative analysis of published peer review literature. Indian J Dent Res. 2012;23(2):293–4. https://doi.org/10.4103/0970-9290.100447 .

Siemens DR, Punnen S, Wong J, Kanji N. A survey on the attitudes towards research in medical school. BMC Med Educ. 2010;10:4. https://doi.org/10.1186/1472-6920-10-4 .

Solomon SS, Tom SC, Pichert J, Wasserman D, Powers AC. Impact of medical student research in the development of physician-scientists. J Investig Med. 2003;51(3):149–56. https://doi.org/10.1136/jim-51-03-17 .

Misztal-Okonska P, Goniewicz K, Hertelendy AJ, et al. How Medical Studies in Poland Prepare Future Healthcare Managers for Crises and Disasters: Results of a Pilot Study. Healthcare (Basel). 2020;8(3):202. https://doi.org/10.3390/healthcare8030202 .

Xu G. On the declaration of educational scientific research topics. Journal of Henan Radio & TV University. 2013;26(01):98–101.

Ju Y, Zhao X. Top three hospitals clinical nurse scientific research ability present situation and influence factors analysis. J Health Vocational Educ. 2022;40(17):125–8.

Zhu Q, Li T, Li X, et al. Industry gain public hospital medical staff performance distribution mode of integration, exploring. J Health Econ Res. 2022;33(11):6-82–6.

Sun YLL. Analysis of hospital papers published based on performance appraisal. China Contemp Med. 2015;22(31):161–3.

Jian Y, Wu J, Liu Y. Citation analysis of seven tertiary hospitals in Yunnan province from 2008 to 2012. Yunnan Medicine. 2014;(6):700–704.

Download references

Acknowledgements

The authors thank all who participated in the nominal group technique and two rounds of the Delphi study.

This study was supported by the National Natural Science Foundation of China (72074160) and the Natural Science Foundation Project of Beijing (9222004).

Author information

Shuyu Liang and Ziyan Zhai contributed equally to this work and joint first authors.

Kai Meng and Yuan Gao contributed equally to this work and share corresponding author.

Authors and Affiliations

Aerospace Center Hospital, No. 15 Yuquan Road, Haidian District, Beijing, 100049, China

Xiaozhi Sun, Jingxuan Jiao & Yuan Gao

School of Public Health, Capital Medical University, No.10 Xitoutiao, Youanmenwai Street, Fengtai District, Beijing, 100069, China

Shuyu Liang, Ziyan Zhai, Xingmiao Feng & Kai Meng

Beijing Tiantan Hospital, Capital Medical University, No. 119 South Fourth Ring West Road, Fengtai District, Beijing, 100070, China

You can also search for this author in PubMed   Google Scholar

Contributions

S.L. and Z.Z. contributed equally to this paper. S.L. took charge of the nominal group technique, data analysis, writing the first draft and revising the manuscript; Z.Z. was responsible for the Delphi survey, data analysis, and writing of the first draft of the manuscript; XF was responsible for the rigorous revision of Delphi methods; X.S. and J.J. were responsible for the questionnaire survey and data collection; Y.G. contributed to the questionnaire survey, organization of the nominal group interview, supervision, project administration and resources; and K.M. contributed to conceptualization, methodology, writing—review; editing, supervision, and project administration. All the authors read and approved the final manuscript.

Corresponding authors

Correspondence to Yuan Gao or Kai Meng .

Ethics declarations

Ethics approval and consent to participate.

This study involved human participants and was approved by the Ethical Review Committee of the Capital Medical University (No. Z2022SY089). Participation in the survey was completely voluntary, and written informed consent was obtained from the participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Liang, S., Zhai, Z., Feng, X. et al. Development of an index system for the scientific literacy of medical staff: a modified Delphi study in China. BMC Med Educ 24 , 397 (2024). https://doi.org/10.1186/s12909-024-05350-0

Download citation

Received : 25 October 2023

Accepted : 26 March 2024

Published : 10 April 2024

DOI : https://doi.org/10.1186/s12909-024-05350-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical staff
• Scientific literacy
• Evaluation indicators

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]