linguistic analysis research paper

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Media
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business Ethics
• Business History
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic History
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Politics and Law
• Public Administration
• Public Policy
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

The Oxford Handbook of Linguistic Analysis

Bernd Heine is Emeritus Professor at the Institut für Afrikanistik, University of Cologne. He has held visiting professorships in Europe, Eastern Asia (Japan, Korea, China), Australia, Africa (Kenya, South Africa), North America (University of New Mexico, Dartmouth College), and South America (Brazil). His 33 books include Possesson: Cognitive Sources, Forces, and Grammaticalization (CUP, 1997); Auxiliaries: Cognitive Forces and Grammaticalization (OUP, 1993); Cognitive Foundations of Grammar (OUP, 1997) (with Tania Kuteva); World Lexicon of Grammaticalization (CUP, 2002); Language Contact and Grammatical Change (CUP, 2005); The Changing Languages of Europe (OUP, 2006), and The Evolution of Grammar (OUP, 2007); and with Heiko Narrog as co-editor, The Oxford Handbook of Linguistic Analysis (OUP, 2011), and The Oxford Handbook of Grammaticalization (OUP, 2012).

Heiko Narrog is professor at Tohoku University, Japan. He received a PhD in Japanese studies from the Ruhr University Bochum in 1997, and a PhD in language studies from Tokyo University in 2002. His publications include Modality in Japanese and the Layered Structure of Clause (Benjamins, 2009), Modality, Subjectivity, and Semantic Change: A Cross-Linguistic Perspective (OUP, 2012), The Oxford Handbook of Linguistic Analysis (OUP, 2010), and The Oxford Handbook of Grammaticalization (OUP, 2011), both co-edited with Bernd Heine.

A newer edition of this book is available.

• Cite Icon Cite
• Permissions Icon Permissions

This handbook compares the main analytic frameworks and methods of contemporary linguistics It offers an overview of linguistic theory, revealing the common concerns of competing approaches. By showing their current and potential applications, the book provides the means by which linguists and others can judge what are the most useful models for the task in hand. Scholars from all over the world explain the rationale and aims of over thirty explanatory approaches to the description, analysis, and understanding of language. Each chapter considers the main goals of the model; the relation it proposes between lexicon, syntax, semantics, pragmatics, and phonology; the way it defines the interaction between cognition and grammar; what it counts as evidence; and how it explains linguistic change and structure.

Signed in as

Institutional accounts.

• GoogleCrawler [DO NOT DELETE]
• Google Scholar Indexing

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code

Institutional access

• Sign in with a library card Sign in with username/password Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Sign in through your institution

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Sign in with a library card

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Welcome to Linguistic Analysis

A peer-reviewed research journal publishing articles in formal phonology, morphology, syntax and semantics. The journal has been in continuous publication since 1976. ISSN: 0098-9053

Please note that Volumes , Issues , Individual Articles , as well as a yearly Unlimited Access Pass (via IP Authentication or Username-and-Password ) to Linguistic Analysis are now available here for purchase and for download on this website. For more information on rates and ordering options, please visit the Rates  page. We will continue to add new material so come back to visit. Please Contact us  if you are interested in specific back issues.

Current Issue

Linguistic Analysis Volume 43 Issues 1 & 2 (2022)

Barcelona Conference on Syntax, Semantics, & Phonology , edited by Anna Paradis & Lorena Castillo-Ros.

This issue brings together a selection of ten papers presented at the 15th Workshop on Syntax, Semantics, and Phonology (WoSSP), held at the Universitat Autònoma de Barcelona, on June 28-29, 2018. WoSSP is a series of on-going workshops organized by PhD students for students who are working in any domain of generative linguistics, and which offers them a forum to share their work in progress . One of the main aims of the WoSSP conference is to provide a space where graduate students who wish to present their work may exchange ideas within different formal approaches to linguistic phenomena.

Read the Introduction

Issues in Preparation

Volume 43, 3-4: Dependency Grammars

This issue, edited by Timothy Osborne, brings together a selection of papers that examine dependency grammars from a variety of perspectives.

Volume 44, 1-2 Pot-pourri

A selection of orthodox and alternate linguistic perspectives, including an in-depth examination of phonology in classical Arabic poetry, and 3 article-length studies of English grammar by Michael Menaugh.

Note: Volume 43, 3-4, will be the last issue of the journal published in paper. Beginning with volume 44, 1-2, all issues will be available in electronic form only on this website <www.linguisticanalysis.com>. Interested parties will be able to purchase single articles, whole issues, or take advantage of the annual All-Access pass to everything.

Note: We are also uploading all past volumes and issues of the journal and expect this process to be completed by the end of 2023.

Thank you for your patience and continued support.

• Architecture and Design
• Asian and Pacific Studies
• Business and Economics
• Classical and Ancient Near Eastern Studies
• Computer Sciences
• Cultural Studies
• Engineering
• General Interest
• Geosciences
• Industrial Chemistry
• Islamic and Middle Eastern Studies
• Jewish Studies
• Library and Information Science, Book Studies
• Life Sciences
• Linguistics and Semiotics
• Literary Studies
• Materials Sciences
• Mathematics
• Social Sciences
• Sports and Recreation
• Theology and Religion
• Publish your article
• The role of authors
• Promoting your article
• Abstracting & indexing
• Publishing Ethics
• Why publish with De Gruyter
• How to publish with De Gruyter
• Our book series
• Our subject areas
• Your digital product at De Gruyter
• Contribute to our reference works
• Product information
• Tools & resources
• Product Information
• Promotional Materials
• Orders and Inquiries
• FAQ for Library Suppliers and Book Sellers
• Repository Policy
• Free access policy
• Open Access agreements
• Database portals
• For Authors
• Customer service
• People + Culture
• Journal Management
• How to join us
• Working at De Gruyter
• Mission & Vision
• De Gruyter Foundation
• De Gruyter Ebound
• Our Responsibility
• Partner publishers

Your purchase has been completed. Your documents are now available to view.

Linguistic Analysis

From data to theory.

• Annarita Puglielli and Mara Frascarelli
• X / Twitter

Please login or register with De Gruyter to order this product.

• Language: English
• Publisher: De Gruyter Mouton
• Copyright year: 2011
• Audience: Researchers, Scholars and Advanced Students of Linguistics concerned with Formal Analysis in a Typological, Comparative Perspective
• Front matter: 8
• Main content: 404
• Published: March 29, 2011
• ISBN: 9783110222517
• Published: March 17, 2011
• ISBN: 9783110222500

SYSTEMATIC REVIEW article

A bibliometric analysis of linguistic research on covid-19.

• 1 Foreign Language Research Department, Beijing Foreign Studies University, Beijing, China
• 2 Center for Linguistics, Literary and Cultural Studies, Sichuan International Studies University, Chongqing, China

Research on COVID-19 has drawn the attention of scholars around the world since the outbreak of the pandemic. Several literature reviews of research topics and themes based on scientometric indicators or bibliometric analyses have already been conducted. However, topics and themes in linguistic-specific research on COVID-19 remain under-studied. With the help of the CiteSpace software, the present study reviewed linguistic research published in SSCI and A&HCI journals to address the identified gap in the literature. The overall performance of the documents was described and document co-citations, keyword co-occurrence, and keyword clusters were visualized via CiteSpace. The main topic areas identified in the reviewed studies ranged from the influences of COVID-19 on language education, and speech-language pathology to crisis communication. The results of the study indicate not only that COVID-19-related linguistic research is topically limited but also that insufficient attention has been accorded by linguistic researchers to Conceptual Metaphor Theory, Critical Discourse Analysis, Pragmatics, and Corpus-based discourse analysis in exploring pandemic discourses and texts.

Introduction

The COVID-19 pandemic has impacted human beings in significant ways, and scientists and researchers have actively responded to the challenges in the post-pandemic era by investigating the phenomenon from the vantage point of their research domains. Since 2020, publications about COVID-19 have proliferated across disciplines. The COVID-19 research literature has also increased in bibliometric and scientometric studies (e.g., Chahrour et al., 2020 ; Deng et al., 2020 ; Colavizza et al., 2021 ), as well as systematic reviews and meta-analyses of a variety of COVID-19 pandemic-related topics, such as the risk factors for critical and fatal COVID-19 cases ( Zheng et al., 2020 ) and considerations of whether asthmatic patients are at higher risk of contracting the virus (e.g., Morais-Almeida et al., 2020 ).

In response to the pandemic, linguistic researchers have provided multilingual public communication services or other helpful language services ( Shen, 2020 ; Di Carlo et al., 2022 ). However, at this juncture, a clear need to map the contributions of the linguistic research community to pandemic literature was in evidence. Hence, the present study reviewed the COVID-19-related literature published in SSCI and A&HCI journals on the Web of Science over the past 2 years to address this need. The study used the CiteSpace bibliometric tool to analyze the current state of linguistic research on COVID-19. CiteSpace is a tool for performing a visual analytic examination of the academic literature of a discipline, a research field, or both, referred to as a knowledge domain ( Chen, 2004 , 2006 , 2020 ). A bibliometric analysis is significant for recognizing the expansion of literature in linguistics. It can aid scholars in gaining quantitative insights into the rise of linguistic research on the COVID-19 pandemic, taking into account the social impact of the disease. The findings can identify the frontiers and gaps in the linguistic study on COVID-19 and guide future research.

Previous studies

The COVID-19 pandemic has exercised a disruptive and profound impact on every aspect of human life. Scientific research papers concerning this pandemic have been growing exponentially. We searched publications related to this topic with “COVID” as the topic term in the Web of Science core collection and got 69,591 results 1 . To help researchers assess the research trends and topics on this issue, several literature surveys have already been implemented. Based on scientometric indicators or bibliometric analyses, these reviews include a focus on research patterns from publications on COVID-19 ( Sahoo and Pandey, 2020 ), the most productive countries and the international scientific collaboration ( Belli et al., 2020 ), and the current hotspots for the disease and future directions ( Zyoud and Al-Jabi, 2020 ). The majority of these studies, however, have concentrated on the medical elements of COVID-19, while paying little attention to the research in the social sciences.

In this context, a recent review by Liu et al. (2022) based on a scientometric analysis of the performance of social science research on COVID-19, covering the landscape, research fields, and international collaborations, represents a notable departure from the prevalent focus of earlier studies. Representing a linguistic focus, another recent study by Heras-Pedrosa et al. (2022) consisted of a systemic analysis of publications in health communication and COVID-19. It found that, in 2020, concepts related to mental health, mass communication, misinformation, and communication risk were more frequently used, and in the succeeding year (2021), vaccination, infodemic, risk perception, social distancing, and telemedicine were the most prevalent keywords.

Within the linguistic field, literature reviews tend to focus on COVID-19-related language education exist. For instance, Moorhouse and Kohnke (2021) explore the lessons learned from COVID-19, and identify and analyze the primary knowledge produced by the English-language teaching community during the epidemic, also offering recommendations for further research on this particular subject. A systemic literature review of adult online learning during the pandemic by Lu et al. (2022) compiled and assessed 124 SSCI literature of empirical studies using a systematic literature review and the literature visualization tool CiteSpace. A bibliometric analysis on “E-learning in higher education in COVID-19” by Brika et al. (2022) deployed VOSviewer, CiteSpace, and KnowledgeMatrix Plus to extract networks and bibliometric indicators about keywords, authors, organizations, and countries. The study offered various insights related to higher education. Distance learning, interactive learning, online learning, virtual learning, computer-based learning, digital learning, and blended learning are among the many terms or subfields of e-learning in higher education.

Linguists have made notable contributions to COVID-19 research. However, there is currently no literature review available on the overall state of the field, including topics such as the most active contributors (e.g., countries, institutions, and journals) to research, dominant topic areas in the field, and trends and gaps in linguistic research. To bridge this gap, this study utilized CiteSpace software 6.1 R2 to conduct a systematic review of the present state of linguistic research on COVID-19. Specifically, this study addressed the following questions:

Q1: Which countries, institutions, and journals have contributed the most to the linguistic research on COVID-19?

Q2: What are the active research areas in the linguistic research on COVID-19?

Q3: What are the recent trends and the research gaps in the linguistic research on COVID-19?

Data collection

As the study was focused on the linguistic field, we searched the Social Science Citation Index (SSCI) and Arts and Humanities Citation Index (A&HCI) available on the Web of Science (WoS) platform. The data were collected through an advanced search. All collected articles/reviews were written in English, and we retrieved the data using the following fields:

1. Topic = (“covid*” OR “*nCoV” OR “SARS-CoV-2” OR “new coronavirus” OR “coronavirus disease 2019” OR “severe acute respiratory syndrome coronavirus-2” OR “novel coronavirus” OR “coronavirus 19”). These terms were only allowed in the title, abstract, or keywords.

2. Time span = 2020–2022

3. Document type = article OR review (the review articles do not include book reviews)

4. (“*”) is a wildcard in WOS that represents any group of characters, including no character.

5. Research area = “linguistics”

Based on the search items listed above, 363 research and review articles were obtained from the Web of Science Core Collection on 25 May 2022. Through manual analysis, the documents completely unrelated to linguistic research, as well as conference abstracts, book reviews, correspondence, and other unrelated documents were excluded. To guarantee the recall ratio, this study used the “remove duplicates (WOS)” function in CiteSpace to filter out duplicated studies from the collected data. After the cleaning procedure, the final dataset contained 355 documents.

The instrument deployed in this study was CiteSpace 6.1 R2 developed by Chen (2004) as a bibliometric analysis tool ( Chen, 2004 , 2006 , 2017 ; Chen et al., 2010 ). The input in this software is a set of bibliographic data files in the field-tagged Institute for Scientific Information Export Format.

In this study, the files were downloaded from the WoS core collection. We chose “full record and cited references” as the record content and the files can be recognized by CiteSpace software directly. When the files are added to the software, they are subjected to the following procedural steps: time slicing, thresholding, modeling, pruning, merging, and mapping (For more details, please see Chen, 2004 ). The outputs of this software are visualized co-citation networks which is to say that each of the networks is presented in a separate interactive window interface. It can show the evolution of a knowledge field on a citation network, display the overall state of a certain field, and highlight some important documents in the development of a field. The strength of CiteSpace lies in the analysis and visualization of the thematic structures and research hotspots. It can provide us with co-citation networks among references, authors, and countries which is of pivotal importance given the research questions underpinning the present study. Hence, to locate important references, recognize research trends, and pinpoint research hotspots in the linguistic research on COVID-19, co-citation documents and keyword co-occurring analyses were conducted in this study through this software.

Global distribution of articles on COVID-19

The overall distribution characteristics are presented below. Figure 1 displays the number of papers published each month since January 2020 when the World Health Organization formally declared the epidemic a global public health emergency. There was only one article about COVID-19 published in January 2020, whereas the publications show a peak in April 2022 with 30 publications. Overall, the results show that publications on the topic are increasing every month. Therefore, we might conclude that linguistic researchers have begun to be increasingly interested in COVID-19 linguistic research.

Figure 1 . Number of articles published by month.

Tables 1 , 2 , respectively, indicate the top 10 most productive countries and institutions for COVID-19 publications. The USA was ranked as the top country in terms of the number of articles related to linguistic publications on COVID-19, with 111 publications in total, followed by China with 57 articles and England with 47 articles ( Table 1 ). In terms of the number of linguistic research publications on COVID-19, Purdue University ranked as the top contributing institution (16 records), followed by the University of London (10 records) and the State University System of Florida (eight publications).

Table 1 . Top 10 most productive countries for COVID-19 linguistic research.

Table 2 . Top 10 most productive institutions for COVID-19 linguistic research.

The 355 articles reviewed in the current study were published in 83 journals. The top 10 most productive journals are listed in Table 3 . System ranked the top journal in the number of published articles, with 21 publications related to COVID-19, followed by American Journal of Speech Language Pathology and International Journal of Language Communication Disorders , with 20 and 18 publications, respectively. As we can see in Table 3 , most of the top 10 journals are related to language education or speech-language pathology.

Table 3 . Top 10 most prolific journals.

Based on the Global Citation Score in the WoS, the top 10 most-cited articles contributing to COVID-19 research are listed in Table 4 . MacIntyre et al. (2020) ranked as the most-cited article with 127 citations. This article is published in System which is also the most productive journal. The top four articles are all about online language teaching during the COVID-19 period.

Table 4 . Top 10 most-cited articles contributing to COVID-19 linguistic research.

Document co-citation analysis

The 355 bibliographic recordings from WoS were visualized and a 1-year time slice was selected for analysis. The size of the node is proportional to the frequency of the cited references. Different colors around nodes represent the frequency of references in different time periods. The labels shown in Figure 2 are all documents with more than three citations, and the connection between nodes shows the co-citation relationship.

Figure 2 . Critical articles in linguistic research on COVID-19.

The top 50 most cited articles every year were selected. There were 176 individual nodes and 562 links, representing cited articles and co-citation relationships among the whole data set, respectively. The results are illustrated in Figure 2 . The results are somewhat different from those obtained from the Global Citation Score in the WoS ( Table 4 ) since the Global Citation Score in the WoS is calculated based on all the citations in WoS, while the document co-citation analysis is based only on the 355 documents retrieved from WoS.

According to the document co-citation analysis, the most co-cited article was written by MacIntyre et al. (2020) . This study explores the issue of language teachers' coping mechanisms and their correlates in the context of the distinctive stressors of the COVID-19 pandemic and the educational responses at the global level. It demonstrates how language teachers have faced a variety of challenges as a result of the global response to the COVID-19 outbreak. High levels of stress have been caused by the quick transition to online education, the blending of job and personal life, and the constant worry about personal and familial wellbeing. With the help of a variety of techniques, teachers were found to be dealing as effectively as they could. Coping strategies that are deemed to be more active and approach-oriented, namely ones that more directly addressed the problems brought on by the phenomenon including the emotions evoked, were found to be connected with more favorable outcomes in terms of psychological health and wellbeing. The greater use of avoidant coping mechanisms was linked to worse psychological outcomes. Increased use of avoidant coping, in particular, was linked to higher stress levels and a range of unpleasant feelings (anxiety, anger, sadness, and loneliness). MacIntyre et al. (2020) also found that a variety of particular techniques were employed by the participants within the approach and avoidant categories of coping, and the majority of them produced outcomes consistent with the category in which they appeared. The multidimensional nature of the stressors required multidimensional coping strategies, but it was obvious that some coping strategies were superior to others. This study by MacIntyre et al. (2020) offers insights into the effectiveness of coping strategies used by language teachers during the crisis and their implications for other stressful events and processes such as school transfers, educational reform, or demanding work periods like the end-of-year exam. MacIntyre et al. (2020) suggest that all pre-service and in-service teacher education programs should incorporate stress management as a fundamental professional competence.

The second most cited article is written by Gacs et al. (2020) , which compares the crisis-prompted online language teaching during the COVID-19 era with well-designed and carefully planned online language education. Due to the 2020 pandemic, many institutions were forced to transition away from face-to-face (F2F) teaching to online instruction. The crisis-prompted online language teaching is different from actual planned online language education. This is because in times of pandemic, war, crisis, natural disaster, or extreme weather, neither teachers nor students are prepared for switching over to online education without good technology literacy, access, and infrastructure. Gacs et al. (2020) describe the process of preparing, designing, implementing, and evaluating online language education when adequate time is available and the concessions one has to make as well when adequate time is not a possibility in times of pandemic or in other emergent conditions. This article presents a roadmap for planning, implementing, and evaluating online education in an ideal and crisis contexts.

The third most cited article conducted by Gao and Zhang (2020) set up a qualitative inquiry to investigate how EFL teachers perceive online instruction in light of their disrupted lesson plans and how EFL teachers teach during the early-stage COVID-19 outbreak developed their information technology literacy. The findings from this study on teachers' perceptions of online instruction during COVID-19 have theoretical ramifications for studies on both teachers' cognitions and online EFL teaching.

It is evident that the three top-cited articles are on the theme of language education. Therefore, it can be concluded that remote online education during a pandemic crisis is the most studied area from the linguistic perspective.

Keyword co-occurrence

In a way, keywords serve as the central summary of articles and serve to convey their major idea and subject matter. The co-occurrence of keywords in an article indicates the degree of closeness between the keywords and the strength of this relationship. According to common perception, the more strongly related two or more terms are, the more often they are likely to appear together. CiteSpace provides a function called Betweenness Centrality to describe the strength. In other words, if a keyword consistently appears alongside other distinct keywords, it is likely that we will see it even if we talk about other related subjects. As a result, the greater the value of Betweenness Centrality a keyword displays, the more significant a keyword is.

A keyword co-occurrence analysis was conducted in this study to identify the research fields and dominant topics. A term analysis of words extracted from keywords was conducted to identify the words or phrases co-occurring in at least two distinct articles. Terms with high frequency may be treated as indicators of hotspots in a certain research field ( Chen, 2004 ). The top five high-frequency keywords were language, student, communication, discourse, and teacher. The keyword co-occurrence network is shown in Figure 3 , and the keywords with frequencies of more than three are displayed in Table 5 .

Figure 3 . Keyword co-occurrence network for documents of linguistic research on COVID-19.

Table 5 . The co-occurring keywords with high frequency.

Cluster interpretations

Based on the analysis of the results of keyword co-occurrence, we used CiteSpace to conduct a cluster analysis. The 355 articles generated 20 clusters in total. Labeling clusters with indexing terms and showing clusters by log-likelihood ratio (LLR), Figure 4 shows the eight most important keyword clusters obtained by keyword co-occurrence analysis. Table 6 shows the keywords lists of the seven important clusters in linguistic research on COVID-19. It illustrates an aggregated distribution in which the most colorful areas overlapped, indicating that these clusters share some basic concepts or information (as suggested by Chen, 2004 ).

Figure 4 . Cluster view of keyword co-occurrence.

Table 6 . Important clusters of keywords in linguistic research on COVID-19.

Cluster #0 is labeled as emergence online teaching

Cluster #6 (online learning) and Cluster #7 (distance learning) are closely related to Cluster #0 since both Cluster #6 (online learning) and Cluster #7 (distance learning) fall under the umbrella of online education during a crisis. Emergency online teaching and online/distance learning are clearly shown to be the focus of linguistic research related to COVID-19.

Due to the COVID-19 pandemic, teaching and learning experienced a shift from physical, in-person (or face-to-face) learning environments to virtual, online learning environments. Although online education is well-established, pandemic-initiated online teaching and learning differed from traditional, well-planned online teaching, thus leading to significant difficulties for both language teachers and students. The stakeholders had to quickly adapt to new environments and learning styles while dealing with the pandemic's personal and societal repercussions on their everyday lives and wellbeing ( MacIntyre et al., 2020 ). The online teaching of foreign and second languages during COVID-19 is referred to as emergency remote teaching (ERT), a term used to describe education temporarily moved online due to unforeseeable events such as natural catastrophes or conflict ( Hodges et al., 2020 ). The difficulties primary school ESOL teachers in the United States encountered as a result of the unexpected instructional adjustments brought on by the COVID-19 epidemic are described by Wong et al. (2022) along with how these difficulties appeared to have impacted the teachers' wellbeing.

There are problems unique to language education, even if English language teachers and students have faced many of the same difficulties as their peers in other disciplines. For instance, many people view the interaction between students and teachers as a crucial component of language acquisition ( Walsh, 2013 ), whereas interaction works very differently in the online mode ( Payne, 2020 ). Therefore, to encourage and support engagement during online language lessons, teachers need to showcase certain competencies ( Cheung, 2021 ; Moorhouse et al., 2021 ).

Understandably, the research community has developed a keen interest on how the COVID-19 pandemic has affected language teaching and learning. More attention is directed toward adapting to the COVID-19 pandemic-initiated online education due to the rapid and abrupt switch from classroom instruction to online learning. For instance, how the students—especially primary pupils—and the teachers adapt to online teaching is the main topic discussed in a special issue of System (2022, volume 105). The COVID-19 pandemic also changed the in-person and on-campus testing into placement testing. Ockey (2021) provides an overview of COVID-19's impact on English language university admissions and placement tests.

Cluster #1 is labeled as science communication

During the COVID-19 pandemic period, it has become very crucial for scientists and government politicians to communicate scientific knowledge to the public to limit the spread of COVID-19. Linguistic factors can play an important role in science communication. A study by Schnepf et al. (2021) inquired into whether complex (vs. simple) scientific statements on mask-wearing could lead audiences to distrust the information and its sources, thus obstructing compliance with behavioral measures communicated on evidence-based recommendations. The study found that text complexity affected audiences inclined toward conspiracy theories negatively. Schnepf et al. (2021) provided recommendations for persuading audiences with a high conspiracy mentality, a group known to be mistrustful of scientific evidence. Janssen et al. (2021) inquired into how the use of lexical hedges (LHs) impacted the trustworthiness ratings of communicators endeavoring to convey the efficacy of mandatory mask-wearing. The study found that scientists were perceived as being more competent and having greater integrity than politicians.

Cluster #2 is labeled as dysphagia

When a society faces a crisis like the COVID-19 pandemic, the impact of COVID-19 on special needs populations, such as people with dysphagia or aphasia or hearing impairments ( Cheng and Cheng, 2022 ; Mathews et al., 2022 ), assumes greater importance for the linguistic community. A study by Jayes et al. (2022) described how UK Speech and language therapists (SLTs) supported differently abled individuals with communication disabilities to make decisions and participate in mental capacity assessments, best interest decision-making, and advance care planning during the COVID-19 pandemic. Govender et al. (2021) investigated how people with a total laryngectomy (PTL) were impacted by COVID-19. Feldhege et al. (2021) conducted an observational study on changes in language style and topics in an online Eating Disorder Community at the beginning of the COVID-19 pandemic. Owing to the severity of the pandemic, speech-language pathologists (SLPs) shifted quickly to virtual speech-language services. Thus, telepractice (cluster #4) also becomes one of the important keyword clusters. Telepractice has been used extensively to offer services to people with communication disorders since the global COVID-19 pandemic. Due to physical separation tactics used to contain the COVID-19 outbreak, many SLPs implemented a live, synchronous online distribution of clinical services. However, SLPs have received synchronous telepractice training to equip them for the shift from an in-person service delivery approach. Using synchronous modes of online clinical practice, Knickerbocker et al. (2021) provide an overview of potential causes of phonogenic voice issues among SLPs in telepractice and suggest prospective preventative techniques to maintain ideal vocal health and function.

Cluster #3 is labeled as social media and it is closely related to Cluster #5 (multilingual crisis communication) since social media research is a way to analyze public communication, particularly during a health crisis. Given the physical restrictions during COVID-19, social media platforms enabled individuals to maintain contact and share ideas. Many studies have investigated the performances of various types of social media platforms during the pandemic, such as Twitter ( Weidner et al., 2021 ), Weibo ( Ho, 2022 ; Yao and Bik Ngai, 2022 ), WhatsApp ( Pérez-Sabater, 2021 ), and YouTube ( Breazu and Machin, 2022 ). Weidner et al. (2021) looked at the characteristics of tweets concerning telepractice via the prism of a well-known framework for using health technology. During the epidemic, there was a surge in telepractice-related tweets. Although several tweets covered ground that is expected in the application of technology, some covered ground that might be particular to speech-language pathology. Yao and Bik Ngai (2022) investigated how People's Daily communicated COVID-19 messages on Weibo. Its findings contribute to the understanding of how public engagement on social media can be augmented via the use of attitudinal messages in health emergencies. Cluster #5 multilingual crisis communication is mostly studied from the perspective of sociolinguistics. Contributing to the sociolinguistics of crisis communication, Ahmad and Hillman (2021) examined the communication strategies employed by Qatar's government in dealing with the COVID-19 pandemic. While a study by Gallardo-Pauls (2021) proposed a specifically linguistic/discursive model of risk communication, Tu et al. (2021) inquired into how pronouns “we” and “you” affected the likelihood to stay at home differently. In another study, Tian et al. (2021) investigated the role of pronouns in crafting supportive messages and hope appeals and facilitating people to cope with COVID-19.

When a society is faced with a crisis, its language can reflect, reveal, and reinforce societal anarchy and divides. A study by Nagar (2021) examined how minority groups—Muslims and migrant workers—experienced marginalization, oppression, and damage through linguistic mechanisms such as silence, presuppositions, accommodations, othering, dog-whistling, and poverty.

Implications for future study

As a discipline, linguistics has contributed significantly to the literature on COVID-19. Based on the results obtained from the above descriptive statistics and visualizations via Citespace, the study found that linguistic research on COVID-19 hitherto has largely focused on the influences of COVID-19 on language education, speech-language pathology, and crisis communication. Language education is one particular strand of applied linguistics, while speech-language pathology and crisis communication, respectively, comprise interdisciplinary studies of language and pathology, and language and communication.

The present state of linguistic research on COVID-19 reveals that there is a dearth of studies deploying linguistic theories such as Conceptual Metaphor Theory, Critical Discourse Analysis, Pragmatics, and Corpus-based discourse analysis. These theories can serve as important heuristics for exploring COVID-19 discourses. A strand of research from the perspective of these theories has highlighted the problematic nature of COVID-19 discourses.

Following the onset of the COVID-19 pandemic, linguists were concerned about the language regarding COVID-19. The Conceptual Metaphor Theory ( Lakoff and Johnson, 1980 ), as one of the primary theoretical constructs in Cognitive Linguistics, was employed by some scholars to explore the COVID-19 discourse. Through their analysis of the conceptual metaphors in different kinds of COVID-19 discourse, linguistic scholars found that the WAR metaphor dominated the COVID-19 discourse ( Bates, 2020 ; Chapman and Miller, 2020 ; Isaacs and Priesz, 2021 ). However, other metaphors such as FIRE remained underexplored concerning the pandemic ( Semino, 2021 ). Although a study by Abdel-Raheem (2021) has explored the multimodal COVID-19 metaphor by examining political cartoons, in general, the multimodal COVID-19 metaphor has not been studied extensively. Further, despite the fact that Preux and Blanco (2021) experimental study explored the influence of the WAR and SPORT domains on emotions and thoughts during the COVID-19 era, the impact of the COVID-19 metaphor on the emotions and mental health of the public has received limited attention.

Critical Discourse Analysis has been deployed by some linguistic researchers. For example, critical discourse analysis was used by Zhang et al. (2021) to compare the reports on COVID-19 and social responsibility expressions in Chinese and American media sources. Based on a case study of U.S. regulations on travel restrictions during the COVID-19 pandemic, Li and Gong (2022) use proximization theory to demonstrate how proximization helps to legitimize health emergency measures. By using a multi-level content analysis technique based on theories of proximization and representation of distant suffering, Florea and Woelfel (2022) investigated the news portrayal of COVID-19 during the year 2020 as proximal vs. remote discourses of suffering. Forchtner and Özvatan, 2022 take a step toward the conceptual integration of narrative (genre) into the Discourse-Historical Approach in Critical Discourse Studies. Their study illuminated the far-right populist Alternative for Germany's (AfD) performances of delegitimization of itself/the nation in relation to Europe and legitimization of itself/the nation by articulating two paradigmatic, transnational crises: climate change and COVID-19. Szabó and Szabó (2022 ) used the discourse dynamics approach to identify the metaphorical terms employed by the Prime Minister to legitimize the crisis management of the Hungarian government and delegitimize critical commentary external to the European Union.

Drawing on critical discourse analysis and textual analysis, Zhou (2021) conducted an interdisciplinary study of the semiotic work dedicated to legitimating Traditional Chinese Medicine (TCM) treatment of COVID-19 in the social media account of an official TCM institution. While CDA analysis of COVID-19 discourses has been undertaken, more CDA-led studies need to be undertaken, given the complexity of power and inequities interwoven reflected in the texts and discourses pertaining to the pandemic.

Pragmatics research on COVID-19 is another underexplored area. Ogiermann and Bella (2021) analyze signs displayed on the doors of closed businesses in Athens and London during the first lockdown of the COVID-19 pandemic, providing some new insights into the dual function of expressive speech acts discussed in pragmatic theory. Blitvich (2022) explores the connections between face-threat and identity construction in the on/off line nexus by focusing on a stigmatized social identity ( Goffman, 1963 ), a local ethnographically specific, cultural position ( Bucholtz and Hall, 2005 ) attributed to some American women stereotypically middle-aged and white who are positioned by others as Karens. Thus, a woman who is perceived to be acting inappropriately, harshly, or in an entitled manner is categorized as a Karen . This incorrect behavior is frequently connected to alleged acts of racism toward minorities. The anti-masker Karens also achieved attention during the COVID-19 pandemic. This research offers a multimodal analysis of a sizable corpus, 256 films of persons whose actions and the way they were seen caused them to be positioned as Karens , to advance our knowledge of the Karen identity. More theories of Pragmatics, such as Relevance Theory, can be employed in the study of COVID-19 discourse.

Corpus-based COVID-19 discourse analysis is also deserving of research attention. Mark Davies has built the Coronavirus Corpus ( https://www.english-corpora.org/corona/ )—an online collection of news articles in English from around the world from January 2020 onwards. The corpus, which was first released in May 2020, currently has about 1,500 million words in size at the cutoff point (16 May 2022), and it continues to grow by three to four million words each day. It can provide vast original discourse data for researchers. For example, based on a 12.3-million-word corpus, Jiang and Hyland (2022) explore keyword nouns and verbs, and frequent noun phrases to understand the central concerns of the public reflected in its news media. In future, more research can be conducted based on the Coronavirus Corpus.

Human life has been greatly affected and disrupted by the COVID-19 pandemic. Scientists and researchers have actively responded to this pandemic by investigating the phenomenon of COVID-19 from the lens offered by their fields of research, and publications relevant to COVID-19 have proliferated rapidly across disciplines since the beginning of 2020. To investigate contributions made by linguistic researchers to pandemic research, the current study carried out a bibliometric analysis of the relevant and available literature. Three hundred and fifty-five bibliometric recordings ranging from January 2020 to May 2022 were collected from WoS, and CiteSpace software was adopted to quantitatively and visually review these papers. The study found that there was continued growth in publications between January 2020 to May 2022. USA was found to be the most productive country in terms of contributions to literature contributing 111 publications pertaining to COVID-19, whereas System ranked as the top journal in the number of published articles related to COVID-19 (21 publications). Through the visualizations of keyword co-occurring analysis and cluster interpretation via Citespace, the study also found that linguistic research on COVID-19 focused largely on the influences of COVID-19 on language education, speech-language pathology, and crisis communication. However, the present review flags the need for more investigations of COVID-19 texts and discourses deploying the explanatory lens of key linguistics theories such as Conceptual Metaphor Theory, Critical Discourse Analysis, Pragmatics, and Corpus-based Discourse Analysis.

Although within its delineated scope, the present study aspired to be as comprehensive as possible, some limitations were unavoidable. For instance, the study searched documents in the Web of Science alone, not including other data sources such as Scopus, Google Scholar, Index Medicus, or Microsoft Academic Search. Further, only one scientometric tool was employed in this review. Future research may make use of a larger database and different analytical tools.

Nonetheless, this study comprises a pioneering review of linguistic research on COVID-19 and identifies and provides a clear overview of international linguistic research in relation to COVID-19. Hence, it can be used as a useful springboard by linguistic researchers interested in probing COVID-19 discourses and texts through the lens of leading theories in the field, thus not only expanding the topical breadth of linguistic research on the pandemic but also generating valuable insights in areas of pragmatics and metaphor as well as CDA and corpus research. These insights are likely to have theoretical as well as practical implications for the field of linguistics.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

This study was funded by the National Social Science Foundation Project (Award Number: 20XYY001, PI: ZH).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1. ^ The search date is 15/05/2022.

Abdel-Raheem, A. (2021). Where Covid metaphors come from: reconsidering context and modality in metaphor. Social Semiotics. 1–40. doi: 10.1080/10350330.2021.1971493. [Epub ahead of print].

CrossRef Full Text | Google Scholar

Ahmad, R., and Hillman, S. (2021). Laboring to communicate: use of migrant languages in COVID-19 awareness campaign in Qatar. Multilingua. 40, 303–337. doi: 10.1515/multi-2020-0119

Alfadda, H., and Mahdi, H. (2021). Measuring students' use of zoom application in language course based on the technology acceptance model (TAM). J. Psycholinguistic Res. 50, 883–900. doi: 10.1007/s10936-020-09752-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Bates, B. R. (2020). The (in)appropriateness of the WAR metaphor in response to SARS-CoV-2: A rapid analysis of Donald J. Trump's rhetoric. Front. Communicat. 5, 1–12. doi: 10.3389/fcomm.2020.00050

CrossRef Full Text

Belli, S., Mugnaini, R., Balt,à, J., and Abadal, E. (2020). Coronavirus mapping in scientifc publications: when science advances rapidly and collectively, is access to this knowledge open to society? Scientometrics. 124, 2661–2685. doi: 10.1007/s11192-020-03590-7

Bischetti, L., Canal, P., and Bambini, V. (2021). Funny but aversive: a large-scale survey of the emotional response to Covid-19 humor in the Italian population during the lockdown. Lingua. 249, 102963. doi: 10.1016/j.lingua.2020.102963

Blitvich, P. (2022). Karen: Stigmatized social identity and face-threat in the on/offline nexus. J. Pragmat. 188, 14–30. doi: 10.1016/j.pragma.2021.11.012

Breazu, P., and Machin, D. (2022). Racism is not just hate speech: ethnonationalist victimhood in YouTube comments about the Roma during Covid-19. Lang. Society . 1–21. doi: 10.1017/S0047404522000070

Brika, S. K. M., Chergui, K., Algamdi, A., Musa., A. A, and Zouaghi, R. (2022). E-Learning research trends in higher education in light of COVID-19: a bibliometric analysis. Front. Psychol. 12, 762819. doi: 10.3389/fpsyg.2021.762819

Bucholtz, M., and Hall, K. (2005). Identity and interaction: a sociocultural linguistic approach. Discourse Stud . 7, 585e614 doi: 10.1177/1461445605054407

Chahrour, M., Assi, S., Bejjani, M., Nasrallah, A. A., Salhab, H., and Fares, M. (2020). A bibliometric analysis of COVID-19 research activity: a call for increased output. Cureus. 12, e7357. doi: 10.7759/cureus.7357

Chapman, C. M., and Miller, D. S. (2020). From metaphor to militarized response: the social implications of “we are at war with COVID-19” – crisis, disasters, and pandemics yet to come. Int. J. Sociol. Social Policy. 40, 1107–1124. doi: 10.1108/IJSSP-05-2020-0163

Chen, C. (2004). Searching for intellectual turning points: progressive knowledge domain visualization. Proc. Natl. Acad. Sci. USA. 101, 5303–5310. doi: 10.1073/pnas.0307513100

Chen, C. (2006). CiteSpace II: detecting and visualizing emerging trends and transient patterns in scientific literature. J. Am. Soc. Inf. Sci. Technol. 57, 359–377. doi: 10.1002/asi.20317

Chen, C. (2017). Science mapping: a systematic review of the literature. J. Data Inf Sci. 2, 1–40. doi: 10.1515/jdis-2017-0006

Chen, C. (2020). A Glimpse of the first eight months of the COVID-19 literature on microsoft academic graph: themes, citation contexts, and uncertainties. Front. Res. Metr. Anal . 5, 607286. doi: 10.3389/frma.2020.607286

Chen, C., Ibekwe-San Juan, F., and Hou, J. (2010). The structure and dynamics of cocitation clusters: a multiple-perspective cocitation analysis. J. Am. Soc. Inf. Sci. Technol. 61, 1386–1409. doi: 10.1002/asi.21309

Cheng, S. Y., and Cheng, S. L. (2022). Parents' psychological stress and their views of school success for deaf or hard-of-hearing children during COVID-19. Communicat. Dis. Q . 1–9. doi: 10.1177/15257401221078788

Cheung, A. (2021). Synchronous online teaching, a blessing or a curse? Insights from EFL primary students'interaction during online English lessons. System. 100, 102566. doi: 10.1016/j.system.2021.102566

Colavizza, G., Costas, R., Traag, V. A., Eck, N. J., Leeuwen, T., and Waltman, L. (2021). A scientometric overview of CORD-19. PLoS ONE. 16, e0244839. doi: 10.1371./journal.pone.0244839

Deng, Z., Chen, J., and Wang, T. (2020). Bibliometric and visualization analysis of human coronaviruses: prospects and implications for COVID-19 research. Front. Cell. Infect. Microbiol . 10, 581404. doi: 10.3389/fcimb.2020.581404

Derakhshan, A., Kruk, K., Mehdizadeh, M., and Pawlak, M. (2021). Boredom in online classes in the Iranian EFL context: sources and solutions. System. 101, 102556. doi: 10.1016/j.system.2021.102556

Di Carlo, P., Lukschy, L., Rey, S., and Vita, V. (2022). Documentary linguists and risk communication: views from the virALLanguages project experience. Linguistics Vanguard . doi: 10.1515/lingvan-2021-0021

Feldhege, J., Moessner, M., Wolf, M., and Bauer, S. (2021). Changes in language style and topics in an online eating disorder community at the beginning of the COVID-19 pandemic: observational study. J. Med. Internet Res. 23, e28346. doi: 10.2196/28346

Florea, S., and Woelfel, J. (2022). Proximal versus distant suffering in TV news discourses on COVID-19 pandemic. Text and Talk. 42, 327–345. doi: 10.1515/text-2020-0083

Forchtner, B., and Özvatan, Ö. (2022). De/legitimising EUrope through the performance of crises The far-right Alternative for Germany on “climate hysteria” and “corona hysteria”. J. Lang. Polit. 21, 208–232. doi: 10.1075/jlp.21064.for

Gacs, A., Goertler, S., and Spasova, S. (2020). Planned online language education versus crisis-prompted online language teaching: lessons for the future. Foreign Lang. Ann . 53, 380–392. doi: 10.1111/flan.12460

Gallardo-Pauls, B. (2021). Riesgos de la comunicación de riesgo: un modelo discursivo para la comunicación de riesgo en emergencias. Círculo de Lingüística Aplicada a la Comunicación. 88, 135-154. doi: 10.5209/clac.77761

Gao, L. X., and Zhang, L. J. (2020). Teacher learning in difficult times: examining foreign language teachers' cognitions about online teaching to tide over COVID-19. Front. Psychol. 11, 549653. doi: 10.3389/fpsyg.2020.549653

Goffman, E. (1963). Stigma. Notes on the Management of the Spoiled Identity . New York: Simon and Schuster Inc.

Google Scholar

Govender, R., Behenna, K., Brady, G., Coffey, M., Babb, M., Patterson, J., et al. (2021). Shielding, hospital admission and mortality among 1216 people with total laryngectomy in the UK during the COVID-19 pandemic: a cross-sectional survey from the first national lockdown. Int. J. Lang. Commun. Disord. 56, 1064–1073. doi: 10.1111/1460-6984.12656

Heras-Pedrosa, C., Jambrino-Maldonado, C., Rando-Cueto, D., and Iglesias-Sánchez, P. P. (2022). COVID-19 study on scientific articles in health communication: a science mapping analysis in web of science. Int. J. Environ. Res. Public Health. 19, 1705. doi: 10.3390/ijerph19031705

Ho, J. (2022). Evading the Lockdown: animal metaphors and dehumanization in virtual space. Metaphor Symbol. 37, 21–38. doi: 10.1080/10926488.2021.1941972

Hodges, C. B., Moore, S., and Lockee, B. B. (2020). The difference between emergency remote teaching and online learning. Educause Review. p. 27. Available onlin at: https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching-and-online-learning

Isaacs, D., and Priesz, A. (2021). COVID-19 and the metaphor of war. J. Paediatr. Child Health. 57, 6–8. doi: 10.1111/jpc.15164

Janssen, I., Hendriks, F., and Jucks, R. (2021). Face masks might protect you from COVID-19: the communication of scientific uncertainty by scientists versus politicians in the context of policy in the making. J. Lang. Soc. Psychol. 40, 602–626. doi: 10.1177/0261927X211044512

Jayes, M., Borrett, S., and Bose, A. (2022). Mental capacity legislation and communication disability: a cross-sectional survey exploring the impact of the COVID-19 pandemic on the provision of specialist decision-making. Int. J. Lang. Commun Disord. 57, 172–181. doi: 10.1111/1460-6984.12685

Jiang, F. K., and Hyland, K. (2022). COVID-19 in the news: the first 12 months. Int. J. Appl. Linguist. 32, 241–258. doi: 10.1111/ijal.12412

Knickerbocker, K., Bryan, C., and Ziegler, A. (2021). Phonogenic voice problems among speech-language pathologists in synchronous telepractice: an overview and recommendations. Semin. Speech Lang. 42, 073–084. doi: 10.1055/s-0040-1722754

Kohnke, L., and Moorhouse, B. L. (2022). Facilitating synchronous online language learning through zoom. RELC J. 53, 296–301. doi: 10.1177/0033688220937235

Lakoff, G., and Johnson, M. (1980). Metaphors We Live by . Chicago: University of Chicago Press.

Li, K., and Gong, X. (2022). Proximization: a critical cognitive analysis of health security discourse. Text and Talk . doi: 10.1515/text-2020-0093

Liu, Y., Yuan, W., and Zhu, S. (2022). The state of social science research on COVID-19. Scientometrics. 127, 369–383. doi: 10.1007/s11192-021-04206-4

Lu, Y., Hong, X., and Xiao, L. (2022). Toward high-quality adult online learning: a systematic review of empirical studies. Sustainability. 14, 2257. doi: 10.3390/su14042257

MacIntyre, P. D., Gregersen, T., and Mercer, S. (2020). Language teachers' coping strategies during the Covid-19 conversion to online teaching: correlations with stress, wellbeing and negative emotions. System. 94, 1–13. doi: 10.1016/j.system.2020.102352

Marler, H., and Ditton, A. (2021). “I'm smiling back at you”: Exploring the impact of mask wearing on communication. Int. J. Lang. Commun. Disord. 56, 205–214. doi: 10.1111/1460-6984.12578

Mathews, E., Cadwell, P., O'Boyle, S., and Dunne, S. (2022). Crisis interpreting and Deaf community access in the COVID-19 pandemic. Perspectives . doi: 10.1080/0907676X.2022.2028873

Moorhouse, B. L., and Kohnke, L. (2021). Responses of the English-Language-Teaching Community to the COVID-19 Pandemic. RELC Journal. 52(3), 359–378. doi: 10.1177/00336882211053052

Moorhouse, B. L., Li, Y., and Walsh, S. (2021). E-Classroom interactional competencies: Mediating and assisting language learning during synchronous online lessons. RELC J . doi: 10.1177/0033688220985274

Morais-Almeida, M., Pit,é, H., Aguiar, R., Ansotegui, I., and Bousquet, J. (2020). Asthma and the coronavirus disease 2019 pandemic: a literature review. Int. Arch. Allergy Immunol. 181, 680–688. doi: 10.1159/000509057

Nagar, I. (2021). The language of suppression: Muslims, migrant workers, and India's response to COVID-19. Lang. Soc. 1–24. doi: 10.1017/S0047404521000816

Ockey, G. J. (2021). An overview of COVID 19's impact on english language university admissions and placement tests. Lang Assess Q . 18, 1–5. doi: 10.1080/15434303.2020.1866576

Ogiermann, E., and Bella, S. (2021). On the dual role of expressive speech acts: Relational work on signs announcing closures during the Covid-19 pandemic. J. Pragmatics. 184, 1–17. doi: 10.1016/j.pragma.2021.07.020

Payne, J. S. (2020). Developing L2 productive language skills online and the strategic use of instructional tools. Foreign Language Annals. 53, 243–249. doi: 10.1111/flan.12457

Pérez-Sabater, C. (2021). Moments of sharing, language style and resources for solidarity on social media: a comparative analysis. J. Pragmat. 180, 266–282. doi: 10.1016/j.pragma.2021.04.034

Piller, I., Zhang, J., and Li, J. (2020). Linguistic diversity in a time of crisis: Language challenges of the COVID-19 pandemic. Multilingua. 39, 503–515. doi: 10.1515/multi-2020-0136

Preux, A., and Blanco, O. (2021). The power of conceptual metaphors in the age of pandemic: The influence of the WAR and SPORT domains on emotions and thoughts. Lang. Communicat. 81, 37–47. doi: 10.1016/j.langcom.2021.08.003

Russell, V. (2020). Language anxiety and the online learner. Foreign Lang. Ann. 53, 338–352. doi: 10.1111/flan.12461

Sahoo, S., and Pandey, S. (2020). Evaluating research performance of Coronavirus and Covid-19 pandemic using scientometric indicators. Online Inf. Rev. 44, 1443–1461. doi: 10.1108/OIR-06-2020-0252

Schnepf, J., Lux, A., Jin, Z., and Formanowicz, M. (2021). Left out—feelings of social exclusion incite individuals with high conspiracy mentality to reject complex scientific messages. J. Lang. Soc. Psychol. 40, 627–652. doi: 10.1177/0261927X211044789

Semino, E. (2021). “Not soldiers but fire-fighters”–metaphors and COVID-19. Health Commun. 36, 50–58. doi: 10.1080/10410236.2020.1844989

Shen, Q. (2020). Commentary: directions in language planning from the COVID-19 pandemic. Multilingua . 39, 625–629. doi: 10.1515/multi-2020-0133

Szabó, P. L., and Szabó, G (2022). Attack of the critics Metaphorical delegitimisation in Viktor Orbán's discourse during the Covid-19 pandemic. J. Lang. Polit. 21, 255–276. doi: 10.1075/jlp.21068.sza

Tian, X., Kim, Y., and Solomon, D. H. (2021). The effects of type of pronouns and hope appeals in supportive messages about COVID-19. J. Lang. Social Psychol. 40, 589–601. doi: 10.1177/0261927X211044721

Tu, K. C., Chen, S. S., and Mesler, R. M. (2021). “We” are in This Pandemic, but “You” can get Through This: the effects of pronouns on likelihood to stay-at-home during COVID-19. J. Lang. Social Psychol. 40, 574–588. doi: 10.1177/0261927X211044799

Walsh, S. (2013). Classroom Discourse and Teacher Development . Edinburgh: Edinburgh University Press. doi: 10.1515/9780748645190

Weidner, K., Lowman, J., Fleischer, A., Kosik, K., Goodbread, P., Chen, B., et al. (2021). Twitter, telepractice, and the COVID-19 pandemic: a social media content analysis. Am. J. Speech Lang. Pathol. 30, 2561–2571. doi: 10.1044/2021_AJSLP-21-00034

Wong, C., Pompeo-Fargnoli, A., and Harriott, W. (2022). Focusing on ESOL teachers' well-being during COVID-19 and beyond. ELT Journal . 76, 1–10. doi: 10.1093/elt/ccab069

Yao, L., and Bik Ngai, C. (2022). Engaging social media users with attitudinal messages during health crisis communication. Lingua , 268, 103199. doi: 10.1016/j.lingua.2021.103199

Zaga, C., Pandian, V., Brodsky, M., Wallace, S., Cameron, T., Chao, C., et al. (2020). Speech-language pathology guidance for tracheostomy during the covid-19 pandemic: an international multidisciplinary perspective. Am. J. Speech-Lang. Pathol. 29, 1320–1334. doi: 10.1044/2020_AJSLP-20-00089

Zhang, Y., Akhtar, N., and Farooq, Q. (2021). Comparative study of chinese and american media reports on the COVID-19 and expressions of social responsibility: a critical discourse analysis. J. Psycholinguist. Res . 51, 455–472. doi: 10.1007/s10936-021-09809-9

Zheng, Z., Peng, F., Xu, B., Zhao, J., Liu, H., and Peng, J. (2020). Risk factors of critical and mortal COVID-19 cases: a systematic literature review and metaanalysis. J. Infect. 81, e16–e25. doi: 10.1016/j.jinf.2020.04.021

Zhou, F. (2021). Traditional Knowledge, science and China's pride: how a TCM social media account legitimizes TCM treatment of Covid-19. Social Semiotics . doi: 10.1080/10350330.2021.1926964

Zyoud, S. E. H., and Al-Jabi, S. W. (2020). Mapping the situation of research on coronavirus disease-19 (COVID-19): a preliminary bibliometric analysis during the early stage of the outbreak. BMC Infect. Dis . 20, 4547–4568. doi: 10.1186/s12879-020-05293-z

Keywords: COVID-19, linguistics, bibliometric analysis, CiteSpace, hot topics

Citation: Peng Z and Hu Z (2022) A bibliometric analysis of linguistic research on COVID-19. Front. Psychol. 13:1005487. doi: 10.3389/fpsyg.2022.1005487

Received: 28 July 2022; Accepted: 22 August 2022; Published: 13 September 2022.

Reviewed by:

Copyright © 2022 Peng and Hu. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Zhibin Peng, zhibin@bfsu.edu.cn

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Help | Advanced Search

Computer Science > Computation and Language

Title: "a good pun is its own reword": can large language models understand puns.

Abstract: Puns play a vital role in academic research due to their distinct structure and clear definition, which aid in the comprehensive analysis of linguistic humor. However, the understanding of puns in large language models (LLMs) has not been thoroughly examined, limiting their use in creative writing and humor creation. In this paper, we leverage three popular tasks, i.e., pun recognition, explanation and generation to systematically evaluate the capabilities of LLMs in pun understanding. In addition to adopting the automated evaluation metrics from prior research, we introduce new evaluation methods and metrics that are better suited to the in-context learning paradigm of LLMs. These new metrics offer a more rigorous assessment of an LLM's ability to understand puns and align more closely with human cognition than previous metrics. Our findings reveal the "lazy pun generation" pattern and identify the primary challenges LLMs encounter in understanding puns.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

AI Index Report

Welcome to the seventh edition of the AI Index report. The 2024 Index is our most comprehensive to date and arrives at an important moment when AI’s influence on society has never been more pronounced. This year, we have broadened our scope to more extensively cover essential trends such as technical advancements in AI, public perceptions of the technology, and the geopolitical dynamics surrounding its development. Featuring more original data than ever before, this edition introduces new estimates on AI training costs, detailed analyses of the responsible AI landscape, and an entirely new chapter dedicated to AI’s impact on science and medicine.

Read the 2024 AI Index Report

The AI Index report tracks, collates, distills, and visualizes data related to artificial intelligence (AI). Our mission is to provide unbiased, rigorously vetted, broadly sourced data in order for policymakers, researchers, executives, journalists, and the general public to develop a more thorough and nuanced understanding of the complex field of AI.

The AI Index is recognized globally as one of the most credible and authoritative sources for data and insights on artificial intelligence. Previous editions have been cited in major newspapers, including the The New York Times, Bloomberg, and The Guardian, have amassed hundreds of academic citations, and been referenced by high-level policymakers in the United States, the United Kingdom, and the European Union, among other places. This year’s edition surpasses all previous ones in size, scale, and scope, reflecting the growing significance that AI is coming to hold in all of our lives.

Steering Committee Co-Directors

Ray Perrault

Steering committee members.

Erik Brynjolfsson

John Etchemendy

Katrina Ligett

Terah Lyons

James Manyika

Juan Carlos Niebles

Vanessa Parli

Yoav Shoham

Russell Wald

Staff members.

Loredana Fattorini

Nestor Maslej

Letter from the co-directors.

A decade ago, the best AI systems in the world were unable to classify objects in images at a human level. AI struggled with language comprehension and could not solve math problems. Today, AI systems routinely exceed human performance on standard benchmarks.

Progress accelerated in 2023. New state-of-the-art systems like GPT-4, Gemini, and Claude 3 are impressively multimodal: They can generate fluent text in dozens of languages, process audio, and even explain memes. As AI has improved, it has increasingly forced its way into our lives. Companies are racing to build AI-based products, and AI is increasingly being used by the general public. But current AI technology still has significant problems. It cannot reliably deal with facts, perform complex reasoning, or explain its conclusions.

AI faces two interrelated futures. First, technology continues to improve and is increasingly used, having major consequences for productivity and employment. It can be put to both good and bad uses. In the second future, the adoption of AI is constrained by the limitations of the technology. Regardless of which future unfolds, governments are increasingly concerned. They are stepping in to encourage the upside, such as funding university R&D and incentivizing private investment. Governments are also aiming to manage the potential downsides, such as impacts on employment, privacy concerns, misinformation, and intellectual property rights.

As AI rapidly evolves, the AI Index aims to help the AI community, policymakers, business leaders, journalists, and the general public navigate this complex landscape. It provides ongoing, objective snapshots tracking several key areas: technical progress in AI capabilities, the community and investments driving AI development and deployment, public opinion on current and potential future impacts, and policy measures taken to stimulate AI innovation while managing its risks and challenges. By comprehensively monitoring the AI ecosystem, the Index serves as an important resource for understanding this transformative technological force.

On the technical front, this year’s AI Index reports that the number of new large language models released worldwide in 2023 doubled over the previous year. Two-thirds were open-source, but the highest-performing models came from industry players with closed systems. Gemini Ultra became the first LLM to reach human-level performance on the Massive Multitask Language Understanding (MMLU) benchmark; performance on the benchmark has improved by 15 percentage points since last year. Additionally, GPT-4 achieved an impressive 0.97 mean win rate score on the comprehensive Holistic Evaluation of Language Models (HELM) benchmark, which includes MMLU among other evaluations.

Although global private investment in AI decreased for the second consecutive year, investment in generative AI skyrocketed. More Fortune 500 earnings calls mentioned AI than ever before, and new studies show that AI tangibly boosts worker productivity. On the policymaking front, global mentions of AI in legislative proceedings have never been higher. U.S. regulators passed more AI-related regulations in 2023 than ever before. Still, many expressed concerns about AI’s ability to generate deepfakes and impact elections. The public became more aware of AI, and studies suggest that they responded with nervousness.

Ray Perrault Co-director, AI Index

Our Supporting Partners

Analytics & Research Partners

Stay up to date on the AI Index by subscribing to the  Stanford HAI newsletter.

This paper is in the following e-collection/theme issue:

Published on 25.4.2024 in Vol 26 (2024)

Leveraging Large Language Models for Improved Patient Access and Self-Management: Assessor-Blinded Comparison Between Expert- and AI-Generated Content

Authors of this article:

Original Paper

• Xiaolei Lv 1, 2, 3, 4, 5, 6 , MSc   ; 
• Xiaomeng Zhang 1, 2, 3, 4, 5, 6 , PhD   ; 
• Yuan Li 1, 2, 3, 4, 5, 6 , MSc   ; 
• Xinxin Ding 1, 2, 3, 4, 5, 6 , PhD   ; 
• Hongchang Lai 1, 2, 3, 4, 5, 6 , Prof Dr Med, PhD   ; 
• Junyu Shi 1, 2, 3, 4, 5, 6 , PhD  

1 Department of Oral and Maxillofacial Implantology, Shanghai PerioImplant Innovation Center, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China

2 College of Stomatology, Shanghai Jiao Tong University, Shanghai, China

3 National Center for Stomatology, Shanghai, China

4 National Clinical Research Center for Oral Diseases, Shanghai, China

5 Shanghai Key Laboratory of Stomatology, Shanghai, China

6 Shanghai Research Institute of Stomatology, Shanghai, China

Corresponding Author:

Junyu Shi, PhD

Department of Oral and Maxillofacial Implantology, Shanghai PerioImplant Innovation Center

Shanghai Ninth People's Hospital

Shanghai Jiao Tong University School of Medicine

Quxi Road No 500

Shanghai, 200011

Phone: 86 21 23271699 ext 5298

Email: [email protected]

Background: While large language models (LLMs) such as ChatGPT and Google Bard have shown significant promise in various fields, their broader impact on enhancing patient health care access and quality, particularly in specialized domains such as oral health, requires comprehensive evaluation.

Objective: This study aims to assess the effectiveness of Google Bard, ChatGPT-3.5, and ChatGPT-4 in offering recommendations for common oral health issues, benchmarked against responses from human dental experts.

Methods: This comparative analysis used 40 questions derived from patient surveys on prevalent oral diseases, which were executed in a simulated clinical environment. Responses, obtained from both human experts and LLMs, were subject to a blinded evaluation process by experienced dentists and lay users, focusing on readability, appropriateness, harmlessness, comprehensiveness, intent capture, and helpfulness. Additionally, the stability of artificial intelligence responses was also assessed by submitting each question 3 times under consistent conditions.

Results: Google Bard excelled in readability but lagged in appropriateness when compared to human experts (mean 8.51, SD 0.37 vs mean 9.60, SD 0.33; P =.03). ChatGPT-3.5 and ChatGPT-4, however, performed comparably with human experts in terms of appropriateness (mean 8.96, SD 0.35 and mean 9.34, SD 0.47, respectively), with ChatGPT-4 demonstrating the highest stability and reliability. Furthermore, all 3 LLMs received superior harmlessness scores comparable to human experts, with lay users finding minimal differences in helpfulness and intent capture between the artificial intelligence models and human responses.

Conclusions: LLMs, particularly ChatGPT-4, show potential in oral health care, providing patient-centric information for enhancing patient education and clinical care. The observed performance variations underscore the need for ongoing refinement and ethical considerations in health care settings. Future research focuses on developing strategies for the safe integration of LLMs in health care settings.

Introduction

Since the launch of ChatGPT by OpenAI [ 1 ] in November 2022, the model has attracted significant global attention, securing over a million users within just 5 days of its release [ 2 ]. ChatGPT is a notable representative of large language models (LLMs), built upon the solid foundation of the GPT architecture [ 3 ]. In today’s technology landscape, other technology giants, including Google and Microsoft, have also developed proprietary and open-source LLMs. These models, pretrained on extensive unlabeled text data sets using self-supervised or semisupervised learning techniques, demonstrate exceptional natural language processing capabilities [ 4 ]. Their advanced capabilities in understanding and generating human-like responses make them particularly relevant for applications in health care, a sector that increasingly relies on digital information and interaction.

The significant potential of such models in the health care sector has captured wide attention among medical professionals [ 5 ]. Notably, without any specialized training or reinforcement, ChatGPT-3.5 performed at or near the passing threshold for the United States Medical Licensing Examination [ 6 ]. This underscores its vast capabilities within medicine, such as retrieving knowledge, aiding clinical decisions, summarizing key findings, triaging patients, and addressing primary care issues. Given its proficiency in generating human-like texts, one of the key applications of LLMs lies in improving health care access and quality through better patient information dissemination.

Early studies have primarily assessed its performance in responding to fundamental questions concerning cardiovascular diseases, cancers, and myopia, yielding encouraging results [ 7 - 10 ]. However, the broader impact of LLMs on patient health care access and quality, particularly in specialized areas such as oral health, has yet to be fully explored. Oral diseases affect over 3.5 billion people worldwide, leading to significant health and economic implications and substantially reducing the quality of life for those affected [ 11 ]. The historical marginalization of oral health care has resulted in considerable gaps in patient literacy, hygiene awareness, and medical consultations [ 12 - 14 ], highlighting a critical area where LLMs could make a significant difference. LLMs have the potential to bridge these gaps by providing accessible, accurate information and advice, thus enhancing patient understanding and self-management. Furthermore, the scarcity of health workers and disparities in resource distribution exacerbate these issues [ 15 , 16 ]. In this context, LLMs, with their rapid advancements, offer a promising avenue for enhancing health care access and quality across various domains [ 17 , 18 ]. A US survey revealed that about two-thirds of adults seek health information on the web and one-third attempt self-diagnosis via search engines [ 19 ]. This trend underscores the growing role of LLMs in digital health interventions [ 20 ], potentially enabling patients to overcome geographical and linguistic barriers in accessing high-quality medical information.

To explore this potential, this study focuses on oral health as an example, assessing the ability of the leading publicly available LLMs, such as Google Bard (Alphabet Inc; subsequently rebranded as Gemini) [ 21 ], ChatGPT-3.5, and ChatGPT-4, in providing patient recommendations for the prevention, screening, and preliminary management of common oral health issues compared to human experts. Both experienced dentists and lay users without medical backgrounds have been invited to evaluate the responses blindly along specified criteria. Our findings are intended to offer valuable insights into the potential benefits and risks associated with using LLMs for addressing common medical questions.

Ethical Considerations

Participants in this study were sourced from our earlier research project, “Bio-bank Construction of Terminal Dentition,” which was approved by the Ethical Committee of Shanghai Ninth People’s Hospital, China (SH9H-2021-T394-2). All participants provided written informed consent prior to the commencement of the study, which clarified their rights to participate and the ability to withdraw from the study at any time. All personal information in this study was anonymized to ensure the privacy and confidentiality of participant data. No compensation was provided to the participants.

Study Design

Figure 1 illustrates the overall study flow diagram. From August 9 to 23, 2023, a questionnaire survey was conducted among outpatients in the Department of Oral and Maxillofacial Implantology at Shanghai Ninth People’s Hospital to inquire about their primary concerns regarding periodontal and implant-related diseases. Informed by the latest consensus reports on periodontal and peri-implant diseases [ 22 ] and clinical experience in tertiary care for periodontology and implantology, our specialist panel (YL, Ke Deng, and Miaoxuan Dai) listed a set of initial questions. Patients rated these on a scale from 0=no concern to 10=extremely concerned and could add any other significant concerns. The questionnaire was administered in Chinese, and the translation and cultural adaptation to English adhered to established guidelines for cross-cultural questionnaire adaptation [ 23 ]. The back translation method was used to ensure both accuracy and cultural appropriateness. After collecting the surveys, the expert panel conducted a thorough review and consolidation process. This involved analyzing patient ratings and comments to identify the most pertinent questions. As a result, a refined set of 40 questions was developed ( Multimedia Appendix 1 ). These questions comprehensively covered 6 domains of periodontal and dental implant-related diseases, including patient education, self-prevention, diagnosis, treatment, management, and support.

From September 4 to 18, 2023, the panel was asked to generate human expert responses to these questions. At the same time, each question was also input into the ChatGPT-3.5, ChatGPT-4, and Google Bard interface, and the subsequent 3 sets of responses were recorded. For the interactions with the LLMs, all responses were generated based on default parameter settings, including temperature and maximum tokens, without any additional specific parameter adjustments. Each question corresponds to a new session and finally has 4 responses. The 4 sets of responses were anonymized and randomly shuffled for evaluation by 5 experienced dentists (JS, Xinyu Wu, Xiaoyu Yu, XZ, and XD) and 5 lay users, respectively, along the axes presented in Multimedia Appendix 2 . The assignment was concealed from the evaluators and outcome examiners (XL and Xue Jiang).

To further understand the stability of responses, each question was submitted to the artificial intelligence (AI) interfaces 3 times from October 28 to 30, 2023. This process was conducted at the same time each day over a 3-day span with constant environmental conditions and model parameters. Each set of 3 responses was independently evaluated by 2 experienced dentists (JS and XL).

Readability Evaluation

The readability of each response was assessed using Readable (Added Bytes), a web-based readability tool featuring multiple indices. Three of the most widely used parameters were recorded. (1) Flesch-Kincaid Grade Level: this is equivalent to the US grade level of education. It shows the required education to be able to understand a text; (2) Gunning Fog Index: this generates a grade level between 0 and 20. It estimates the education level required to understand the text. Text aimed at the public should aim for a grade level of around 8. Text above a 17 requires a graduate-level understanding; and (3) Flesch Reading Ease Score: this is between 1 and 100, with 100 being the highest readability score. Scoring between 70 and 80 is equivalent to school grade level 8. This means text should be fairly easy for the average adult to read.

Dentist Evaluation

Five experienced dentists (JS, Xinyu Wu, Xiaoyu Yu, XZ, and XD) were asked to grade each response using the visual analog scale (VAS) along the following 3 axes ( Multimedia Appendix 2 ). The final results were synthesized based on the scores from all 5 dentists. The average was calculated after removing extreme values to minimize errors.

• Appropriateness: How appropriate is this answer to the patient asking? (0=totally inappropriate or incorrect and 10=totally appropriate)
• Comprehensiveness: Does the answer omit any content it should not? (0=yes, great clinical significance and 10=no missing content)
• Harmlessness: What is the extent and likelihood of possible harm? (0=severe harm and 10=no harm)

Lay User Evaluation

Five lay users were also asked to grade each response using the VAS along the following 2 axes ( Multimedia Appendix 2 ). Final results were synthesized based on the scores from all 5 lay users, and the average was calculated after removing extreme values.

• Intent capture: How well does the answer address the intent of the question? (0=does not address query and 10=addresses query)
• Helpfulness: How helpful is this answer to the user? (0=not helpful at all and 10=very helpful)

Further Evaluation of LLMs in Different Conditions and Domains

To further investigate whether the responses of LLMs differ across various conditions and domains, detailed subanalyses were conducted on 2 oral issues (periodontitis and dental implant) and 6 medical care domains (patients’ education, prevention, diagnosis, treatment, management, and support).

Stability Evaluation

Each question was submitted to the AI interfaces 3 times, and the responses were recorded. Two experienced dentists (JS and XL) independently evaluated each set of 3 responses. Responses were graded as “correct” or “incorrect” based on clinical judgment and the content or as “unreliable” if the 3 responses were inconsistent. Any set with at least 1 incorrect response was graded as incorrect.

Statistical Analysis

Statistical analyses were conducted using SAS software (version 9.4; SAS Institute) and GraphPad Prism 9 (GraphPad Software, Inc). Quantitative data of normal distribution were summarized as means and SDs. Intraclass correlation coefficient (ICC) was used to access interrater agreement. Repeated measures ANOVA was used to compare scores across the LLMs and human experts. Additionally, paired chi-square tests were used to assess the stability of AI responses. Statistical significance was set at a P <.05.

Readability Evaluation Results

In the readability evaluation, detailed in Table 1 and Figure 2 , Google Bard was found to be the most readable for the public. It scored the lowest on Flesch-Kincaid Grade Levels (mean 7.86, SD 0.96) and Gunning Fog Index (mean 9.62, SD 1.11) and the highest on the Flesch Reading Ease Score (mean 61.72, SD 6.64), indicating it was easier to comprehend and had superior readability (all P <.001). Furthermore, the word count for all 3 LLMs, averaging over 300 words, was significantly higher than the approximately 100 words typical for human experts.

a Flesch-Kincaid Grade and Gunning Fog Index show the education level needed for understanding; a lower score means that it is easier.

b Flesch Reading Ease Scores from 1 to 100, with a higher score meaning easier to read.

Dentist Evaluation Results

Table 2 and Figure 3 present the evaluation results of dentists. Google Bard demonstrated significantly lower appropriateness score than human experts (mean 8.51, SD 0.37 vs mean 9.60, SD 0.33; P =.03), while ChatGPT-3.5 and ChatGPT-4 got comparable scores (mean 8.96, SD 0.35 and mean 9.34, SD 0.47, respectively). Google Bard also showed a great level of missing content than ChatGPT-3.5 (mean 8.40, SD 0.60 vs mean 9.46, SD 0.14; P =.04). No other difference of comprehensiveness was significant between groups. All 3 LLMs showed superior harmlessness scores comparable with human experts (Google Bard: mean 9.34, SD 0.11; ChatGPT-3.5: mean 9.65, SD 0.20; ChatGPT-4: mean 9.69, SD 0.41; and human experts: mean 9.68, SD 0.4, out of a maximum score of 10). The ICC indicated “substantial” agreement among dentists with a value of 0.715.

Lay User Evaluation Results

Table 2 and Figure 4 display the evaluation results of lay users. No significant difference between the responses of LLMs and human experts, with both effectively capturing user intent and providing helpful answers for them (all P >.05). The ICC indicated “moderate” agreement among lay users with a value of 0.586.

Subanalysis Results

Subanalyses were conducted across the 2 oral issues and 6 medical care domains. In periodontal questions, Google Bard still demonstrated significantly lower appropriateness than human experts ( P =.04). In implant questions, Google Bard performed less appropriately than ChatGPT-4 and human experts ( P =.03 and P =.01, respectively) and less comprehensively than ChatGPT-3.5 and 4 ( P =.02 and P =.05, respectively). All 3 LLMs performed consistently well in harmlessness across 6 medical care domains. In terms of appropriateness and comprehensiveness, all 3 LLMs achieved comparable VAS scores with human experts in the “prevention” and “treatment” domains. In the “education,” “diagnosis,” “management,” and “support” domains, 2 ChatGPT models achieved comparable scores, while Google Bard was significantly less appropriate than human experts ( P =.01, P =.02, P =.04, and P =.03, respectively). Consistently, Google Bard omits more content than 2 ChatGPT models and human experts in these domains. What is more, in terms of intent capture, Google Bard performed better in the domains of “prevention,” “management,” and “support” than in the “diagnosis.” Detailed subanalyses are shown in Multimedia Appendices 3 and 4 .

Stability Evaluation Results

Table 3 presents the stability evaluation results. All 3 AI models answered 40 questions, except Google Bard, which did not answer the question “Is dental implant surgery painful?” in 2 of 3 attempts. ChatGPT-4 achieved the highest number of correct answers (n=34, 85%), the fewest incorrect answers (n=4, 10%), and the fewest unreliable answers (n=2, 5%). ChatGPT-3.5 had more correct responses than Google Bard (n=29, 72% vs n=25, 62%) but also recorded more incorrect responses (n=8, 20% vs n=7, 17%). Moreover, ChatGPT-3.5 had fewer unreliable responses compared to Google Bard (n=3, 7% vs n=8, 20%).

Principal Findings

This study critically evaluates the use of LLMs AI such as Google Bard, ChatGPT-3.5, and ChatGPT-4 in the context of patient self-management for common oral diseases, drawing a comparative analysis with human expert responses [ 24 ]. Our findings reveal a multifaceted landscape of the potential and challenges of integrating LLMs into health care. The results underscore a promising future for AI chatbots to assist clinical workflows by augmenting patient education and patient-clinician communication around common oral disease queries with comparable accuracy, harmfulness, and comprehensiveness to human experts. However, they also highlight existing challenges that necessitate ongoing optimization strategies since even the most capable models have some inaccuracy and inconsistency.

Comparison to Prior Work

In the comprehensive evaluation of the 3 LLMs, ChatGPT-4 emerged as the superior model, consistent with prior assessments in various medical domains [ 10 , 25 , 26 ]. This superior performance is likely attributable to its substantially larger training data set, continuous architectural enhancements, and notable advancements in language processing, contextual comprehension, and advanced reasoning skills [ 20 ]. These improvements are crucial in health care applications, where the precision and relevance of information are critical. Interestingly, despite ChatGPT-4 showing greater stability, no significant differences were observed between ChatGPT-4 and ChatGPT-3.5 in dentist and patient evaluations. Given that ChatGPT-4 is a premium version not universally accessible, ChatGPT-3.5 holds significant value for broader applications.

In assessments spanning both periodontal and implant-related issues as well as a range of medical domains, Google Bard consistently demonstrated the least effective performance in addressing basic oral disease queries, particularly within the “diagnosis” domain. Notably, Google Bard’s tendency to avoid questions about dental implant surgery pain, in contrast to ChatGPT’s consistent responsiveness, might reflect differing strategies in risk management. However, in terms of readability, an important criterion for nonmedical users’ educational materials, Google Bard outperformed even human experts. This aligns with prior studies assessing LLMs’ readability and agrees with the impact of different training data and preprocessing methods on LLMs’ readability [ 27 , 28 ].

Future Directions

Moreover, all 3 LLM chatbots performed similarly in providing harmless responses. In the context of medical conversation, these AI models consistently encouraged patients to seek professional medical advice, underscoring the irreplaceable role of human expertise diagnosis and treatment. However, the results of the lay user evaluation warrant caution, as they show that AI models were comparable to human experts in intent capture and helpfulness. This ambiguous distinction poses a paradox. On one hand, it suggests user acceptance in AI-provided information, underscoring their capability to effectively address user inquiries. On the other hand, it discreetly underscores a potential risk: the lay users’ limited ability to judge the accuracy of complex medical information, which might inadvertently lead to AI disseminating misconceptions or inappropriate guidance. This underscores the critical need to address the ethical consideration of integrating AI in health care [ 29 , 30 ]. It is essential to clearly define the responsibilities and risks associated with using AI in patient education and in facilitating patient-clinician communication.

The observed performance differences among the AI models, influenced by factors like diverse training data sets and algorithmic updates, combined with the lay evaluations, emphasize the importance of customizing and continually updating LLMs for oral health care. Tailoring AI to meet specific oral health needs and maintaining current medical standards are crucial to ensure safe and accurate patient support.

Strengths and Limitations

LLMs demonstrate varied performances across different medical fields, which can be attributed to the varying depth of available web-based data on each topic. It is imperative to thoroughly evaluate their efficacy across diverse medical topics. In comparison to systemic diseases, using LLMs for basic oral health conditions offers substantial benefits. First, the narrower scope of oral diseases renders personalized oral hygiene advice and disease risk prediction via AI more viable. Second, the relative simplicity of oral structures, combined with AI’s advanced image recognition capabilities, facilitates the more feasible identification and analysis of oral imagery, thus aiding early-stage problem detection. This research underscores the potential of using AI to provide individualized oral health guidance to patients, which could significantly broaden their access to medical knowledge, reduce health care expenses, enhance medical efficiency, lower public health costs, balance medical resource distribution, and relieve national economic burdens.

To our knowledge, this is the first study to evaluate the application of current LLMs comprehensively and rigorously in basic oral diseases. The robust experimental design and the implementation of blinding largely reduce evaluator bias, ensuring the validity of the results. However, this study is not without limitations. First, its methodology, based on simulated question-and-answer scenarios, does not fully replicate real-world clinical interactions. Future research should involve actual patient interactions for more accurate assessment. Second, the performance of the LLM largely depends on the quality of the prompt guiding the model, highlighting the necessity for further research in this area. With the currently rapid evolution of LLMs, there is a critical need to develop specialized chatbots with medical expertise, combining the strengths of current LLMs for health care applications. Currently, integrating medical professionals seems to be the most effective strategy for optimizing AI applications in health care.

Conclusions

LLMs, particularly ChatGPT-4, demonstrate promising potential in providing patient-centric information for common oral diseases. Variations in performance underscore the need for ongoing refinement and ethical considerations. Future studies should explore strategies to integrate LLMs effectively in health care settings, ensuring their safe and effective use in patient care.

Conflicts of Interest

None declared.

Question list.

Evaluation axes.

Subanalysis results of periodontal and implant-related queries.

Subanalysis results of 6 medical care domains.

• ChatGPT. OpenAI. URL: https://chat.openai.com/ [accessed 2024-04-22]
• Schulman J, Zoph B, Kim C. ChatGPT: optimizing language models for dialogue. OpenAI blog. 2022. URL: https://chatgpt.r4wand.eu.org/# [accessed 2022-04-12]
• Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, et al. A survey of large language models. Preprint posted online on November 24, 2023. [ FREE Full text ]
• Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training. Amazon AWS. URL: https://www.mikecaptain.com/resources/pdf/GPT-1.pdf
• Haupt CE, Marks M. AI-generated medical advice-GPT and beyond. JAMA. 2023;329(16):1349-1350. [ CrossRef ] [ Medline ]
• Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sarraju A, Bruemmer D, Van Iterson E, Cho L, Rodriguez F, Laffin L. Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model. JAMA. 2023;329(10):842-844. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Haver HL, Ambinder EB, Bahl M, Oluyemi ET, Jeudy J, Yi PH. Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT. Radiology. 2023;307(4):e230424. [ CrossRef ] [ Medline ]
• Rahsepar AA, Tavakoli N, Kim GHJ, Hassani C, Abtin F, Bedayat A. How AI responds to common lung cancer questions: ChatGPT vs Google Bard. Radiology. 2023;307(5):e230922. [ CrossRef ] [ Medline ]
• Lim ZW, Pushpanathan K, Yew SME, Lai Y, Sun CH, Lam JSH, et al. Benchmarking large language models' performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine. 2023;95:104770. [ FREE Full text ] [ CrossRef ] [ Medline ]
• GBD 2017 Disease and Injury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet. 2018;392(10159):1789-1858. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Peres MA, Macpherson LMD, Weyant RJ, Daly B, Venturelli R, Mathur MR, et al. Oral diseases: a global public health challenge. Lancet. 2019;394(10194):249-260. [ CrossRef ] [ Medline ]
• Global periodontal health: challenges, priorities and perspectives. FDI World Dental Federation. URL: https://www.fdiworlddental.org/global-periodontal-health-challenges-priorities-and-perspectives [accessed 2024-04-02]
• White Paper on Prevention and Management of Periodontal Diseases for Oral Health and General Health. FDI World Dental Federation. URL: https:/​/www.​fdiworlddental.org/​resource/​white-paper-prevention-and-management-periodontal-diseases-oral-health-and-general-health [accessed 2024-04-02]
• Watt RG, Daly B, Allison P, Macpherson LMD, Venturelli R, Listl S, et al. Ending the neglect of global oral health: time for radical action. Lancet. 2019;394(10194):261-272. [ CrossRef ] [ Medline ]
• Tonetti MS, Jepsen S, Jin L, Otomo-Corgel J. Impact of the global burden of periodontal diseases on health, nutrition and wellbeing of mankind: a call for global action. J Clin Periodontol. 2017;44(5):456-462. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fatani B. ChatGPT for future medical and dental research. Cureus. 2023;15(4):e37285. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Eggmann F, Weiger R, Zitzmann NU, Blatz MB. Implications of large language models such as ChatGPT for dental medicine. J Esthet Restor Dent. 2023;35(7):1098-1102. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kuehn BM. More than one-third of US individuals use the internet to self-diagnose. JAMA. 2013;309(8):756-757. [ CrossRef ] [ Medline ]
• Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N Engl J Med. 2023;388(13):1233-1239. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gemini. Google. URL: https://gemini.google.com
• Caton JG, Armitage G, Berglundh T, Chapple ILC, Jepsen S, Kornman KS, et al. A new classification scheme for periodontal and peri-implant diseases and conditions—introduction and key changes from the 1999 classification. J Clin Periodontol. 2018;45(Suppl 20):S1-S8. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gjersing L, Caplehorn JRM, Clausen T. Cross-cultural adaptation of research instruments: language, setting, time and statistical considerations. BMC Med Res Methodol. 2010;10:13. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Perlis RH, Fihn SD. Evaluating the application of large language models in clinical research contexts. JAMA Netw Open. 2023;6(10):e2335924. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ali R, Tang OY, Connolly ID, Fridley JS, Shin JH, Sullivan PLZ, et al. Performance of ChatGPT, GPT-4, and Google Bard on a neurosurgery oral boards preparation question bank. Neurosurgery. 2023;93(5):1090-1098. [ CrossRef ] [ Medline ]
• Holmes J, Liu Z, Zhang L, Ding Y, Sio TT, McGee LA, et al. Evaluating large language models on a highly-specialized topic, radiation oncology physics. Front Oncol. 2023;13:1219326. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Seth I, Lim B, Xie Y, Cevik J, Rozen WM, Ross RJ, et al. Comparing the efficacy of large language models ChatGPT, BARD, and Bing AI in providing information on rhinoplasty: an observational study. Aesthet Surg J Open Forum. 2023;5:ojad084. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhu L, Mou W, Chen R. Can the ChatGPT and other large language models with internet-connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge? J Transl Med. 2023;21(1):269. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jobin A, Ienca M, Vayena E. The global landscape of AI ethics guidelines. Nat Mach Intell. 2019;1(9):389-399. [ CrossRef ]
• Price WN, Cohen IG. Privacy in the age of medical big data. Nat Med. 2019;25(1):37-43. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by G Eysenbach, T de Azevedo Cardoso; submitted 27.12.23; peer-reviewed by L Weinert, L Zhu, W Cao; comments to author 26.02.24; revised version received 04.03.24; accepted 19.03.24; published 25.04.24.

©Xiaolei Lv, Xiaomeng Zhang, Yuan Li, Xinxin Ding, Hongchang Lai, Junyu Shi. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 25.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Linguistic and semantic characteristics of articles and peer review reports in Social Sciences and Medical and Health Sciences: analysis of articles published in Open Research Central

• Open access
• Published: 03 July 2023
• Volume 128 , pages 4707–4729, ( 2023 )

Cite this article

You have full access to this open access article

• Andrijana Perković Paloš   ORCID: orcid.org/0000-0003-3048-2023 1   na1 ,
• Antonija Mijatović   ORCID: orcid.org/0000-0003-1733-582X 1   na1 ,
• Ivan Buljan   ORCID: orcid.org/0000-0002-8719-7277 1 , 2 ,
• Daniel Garcia-Costa   ORCID: orcid.org/0000-0002-8939-8451 3 ,
• Elena Álvarez-García 3 ,
• Francisco Grimaldo   ORCID: orcid.org/0000-0002-1357-7170 3 &
• Ana Marušić   ORCID: orcid.org/0000-0001-6272-0917 1  

1378 Accesses

Explore all metrics

We aimed to examine the differences in articles, peer review and editorial processes in Medical and Health Sciences vs. Social Sciences. Our data source was Open Research Central (ORC) portal, which hosts several journal platforms for post-publication peer review, allowing the analysis of articles from their submission, regardless of the publishing outcome. The study sample included 51 research articles that had Social Sciences tag only and 361 research articles with Medical and Health Sciences tag only. Levenshtein distance analysis showed that text changes over article versions in social science papers were statistically significant in the Introduction section. Articles from Social Sciences had longer Introduction and Conclusion sections and higher percentage of articles with merged Discussion and Conclusion sections. Articles from Medical and Health Sciences followed the Introduction-Methods-Results-Discussion (IMRaD) structure more frequently and contained fewer declarations and non IMRaD sections, but more figures. Social Sciences articles had higher Word Count, higher Clout, and less positive Tone. Linguistic analysis revealed a more positive Tone for peer review reports for articles in Social Sciences and higher Achievement and Research variables. Peer review reports were significantly longer for articles in Social Sciences but the two disciplines did not differ in the characteristics of the peer review process at all stages between the submitted and published version. This may be due to the fact that they were published on the same publication platform, which uses uniform policies and procedures for both types of articles.

Similar content being viewed by others

Why, When, Who, What, How, and Where for Trainees Writing Literature Review Articles

Ethical Considerations of Conducting Systematic Reviews in Educational Research

How to design bibliometric research: an overview and a framework proposal

Avoid common mistakes on your manuscript.

Introduction

Research disciplines differ not only in the topic and focus of their research but also in the structure of the manuscripts, peer review evaluation, editorial processes, and research methodology. Because research work is more closely linked to reading and writing in some areas, such as humanities, the differences may occur in the language and writing styles and in the preferred ways of communicating research findings, whether it is in the form of an article or a monograph. Bellow we present literature review on the differences in manuscripts and peer-review processes across different research areas.

Manuscript differences

A study of linguistic differences between research areas in 500 abstracts of research articles published in 50 high-impact journals, showed that each of the five areas—earth, formal (i.e. related to formal systems such as logic, mathematics, statistics), life, physical and social science—have their own set of “macro-structural, metadiscoursal and formulation features” (Ngai et al., 2018 ). For example, medical and health sciences have been using the IMRaD article format (Introduction, Methods, Results, Discussion) since the 1950s (Sollaci et al., 2004 ), whereas research disciplines such as social sciences tend to have a more flexible article structure. Predominant publication outputs in social sciences (and also humanities) are academic monographs and books (Williams et al., 2009 ; Wolfe, 1990 ). Also, articles in social sciences journals tend to be longer than those in medical journals (Silverberg & Ray, 2018 ). Another difference is that the natural sciences research strategies are more adapted to “large concentrated knowledge clusters”, whereas social sciences usually adapt their research strategies to “many small isolated knowledge clusters” (Jaffe, 2014 ).

Peer review differences

The importance of the evaluation process for researchers and research in general has been the topic of numerous studies, from examining the impact of peer review on submitted manuscripts to some specific characteristics of peer review reports. Recently, initiatives to share peer review data on peer review (Squazzoni et al., 2020 ), have brought about better understanding of the peer review process across different disciplines (Buljan et al., 2020 ; Squazzoni et al., 2021a , 2021b ), particularly in the type of peer review. In open peer review, authors and reviewers know each other’s identity and sometimes reviewer reports are published next to the articles (Ross-Hellauer, 2017 ), whereas in post-publication peer review, articles are reviewed after publication in an open review process (Ford, 2015 ). In medical and health Sciences, open and post-publication peer review are becoming more common and peer review reports are often published together with the articles (Hamilton et al., 2020 ). In social sciences, the peer review process has remained closed because double blind peer review is still preferred (Karhulahti & Backe, 2021 ). In recent years, however, some platforms publishing articles from social sciences and humanities, such as Palgrave Macmillan, have adopted the practice of open peer review (Palgrave MacMillan, 2014 ). This allows researchers to study the peer review process in social sciences and humanities and to compare it with other research areas.

A study exploring the role of peer review in increasing the quality and value of manuscripts (Garcia-Costa et al., 2022 ) showed that the impact of peer review is shared across research areas but not without certain differences, as reports from social sciences and economic journals displayed the highest “developmental standards”.

Regarding the linguistic differences, a study of almost half a million peer review reports from 61 journals (Buljan et al., 2020 ) showed that peer review reports were longer in social sciences than in medical journals, but there were no differences in the length between double- and single-blind reviews. Language characteristics were also different across disciplines (Buljan et al., 2020 ): peer review reports in medical journals had low Authenticity (impersonal and cautious language) and high analytical Tone (use of more formal and logical language), whereas the language of peer review reports in social sciences journals had high Authenticity (personal and open language) and high Clout (honest and humble reporting and high level of confidence). Using natural language techniques, Rashidi et al. ( 2020 ), studied published articles and their open peer review reports from F1000Research journal, which uses a post-publication peer review. They found consistency and similarity in the use of salient words, like those from the Medical Subject Headings (MeSH) of MEDLINE. F1000Research platform was also used to develop a sentiment analysis program to detect praise and criticism in peer evaluations (Thelwall et al., 2020 ), which showed that negative evaluations in reviewer’s comments better predict review outcomes than positive comments.

Peer review research carries major challenges due to the lack of access to the whole process of scientific publication. This means that peer review process remains hidden from the submission of the article to its rejection or publication after peer review, hindering our understanding of the publishing process. For this reason, we decided to use the Open Research Central (ORC) portal, which hosts several journal platforms for post-publication peer review (Tracz, 2017 ) to study the characteristics of articles and peer review reports in Medical and Health Sciences and Social Sciences. Journals at the ORC platform are multidisciplinary and use the post-publication review: the articles are publicly available upon submission, and access is possible to the whole peer review and editorial decision-making process (ORC, 2022 ). To our knowledge, there has not been a study that analysed the peer review reports from this platform. The aim of our study was to examine possible differences in the submitted articles and peer review process in Medical and Health Sciences vs. Social Sciences. We examined: (i) the structural and linguistic differences between research articles; (ii) the characteristics of the peer review process; (iii) the language of peer review reports; and (iv) the outcomes of peer review process.

Data source: ORC portal

ORC currently includes the following journals: F1000Research, Wellcome Open Research, Gates Open Research, MNI Open Research, HRB Open Research, AAS Open Research, AMRC Open Research and Emerald Open Research.

Identify the articles: get articles’ DOIs

Using the ORC search engine ( https://openresearchcentral.org/browse/articles ) and Python 3.8.5 ( https://www.python.org/downloads/release/python-385/ ), we performed 2 automatic queries applying the following filters: “Article type(s): Research Article” and “Subject area: Medical and Health Sciences”, or “Subject area: Social Science”. We then extracted the articles’ DOIs using requests ( https://docs.python-requests.org/en/latest/ ) and Beautiful Soup ( https://beautiful-soup-4.readthedocs.io/en/latest/ ) HTTP libraries for Python. We retrieved 1912 Medical and Health and 477 Social Sciences articles. In order to create the samples of articles with clear Medical and Health vs. Social Sciences content, we excluded articles with a tag for both disciplinary fields, those with a tag for Medicine and Health Sciences and any other disciplinary field except Biology and Life Science, and those with a tag both for Social Sciences and Biology and Life Sciences. This was done also to ensure that manuscripts and peer review reports were not influenced by the language and writing style of multiple research areas. This left with 408 Medical and Health and 54 Social Science articles (Fig.  1 ).

A flowchart representing methods workflow (created using Zen Flowchart: https://www.zenflowchart.com/ )

Retrieve the articles in XML format

Using the DOIs of filtered articles, we downloaded the articles manually in an XML format. We used the XML article format in order to achieve better quality data mining due to its semantic and machine-readable tagging. All versions of articles were downloaded in order to get complete article information.

XML Parser: extract and save relevant data

We used the ElementTree library in Python (xml.etree.ElementTree — The ElementTree XML API — Python 3.10.1 documentation) for parsing data from the XML files. First, the articles that had not been reviewed were excluded, yielding a total of 51 articles with a Social Sciences tag and 361 articles with a Medical and Health Sciences tag. A simplified sequence diagram for XML document parser is shown in Fig.  2 .

A simplified sequence diagram for ORC XML parser (the diagram was created using PlantUML open-source tool https://plantuml.com/sequence-diagram )

Using the scripts, we extracted the following variables:

Length of an article and the length of the individual article chapter – Introduction, Methods, Results and Discussion (IMRaD);

Number of figures, tables and supplementary material in the articles;

Percent of articles following the IMRaD structure;

Linguistic characteristics of the articles such as Tone, Sentiment, etc.;

Male to female ratio among article reviewers;

Time for an article to be first posted;

Number of rounds of review until the article is accepted;

Time to review each version of the article;

Time for an article to have a “positive” status;

Length of review comments;

Linguistic characteristics of research articles and corresponding peer reviews; and

Reviewers’ recommendations.

Reviewers’ gender was determined by using Python class Genderize from Genderize.io web service ( https://pypi.org/project/Genderize/ ), which predicts the gender of a person given their name.

Details on how each of the variables was extracted from the XML files can be found in the following Python scripts: https://github.com/Tonija/ORC_scripts .

The results of the scripts were saved in a csv table and used for linguistic and statistical analysis.

Linguistic analysis

Linguistic inquiry word count (liwc).

The texts of the articles and corresponding peer reviews were analysed using the Linguistic Inquiry Word Count (LIWC) text analysis software program (Pennebaker et al., 2015a ). We calculated LIWC’s five default variables (Word count, Analytic, Clout, Authentic, Tone), where Word count (WC) is the raw number of words in a given text, while Analytic, Clout, Authentic, and Tone are the linguistic variables expressed as percentages of total words within a text (Pennebaker et al., 2015b ). Higher scores on the Analytic dimension describe the use of formal, logical, and hierarchical language; higher Clout score refers to a higher level of leadership and confidence; higher Authentic score points to a more personal way of writing; and higher Tone score represents a more positive emotion dimensions (Pennebaker et al., 2015b ).

We also analysed seven other LIWC categories related to research evaluation, used for linguistic study of letters of recommendation for academic job applicants (Schmader et al., 2007 ), and for text analysis of research grant reviewers’ critiques (Kaatz et al., 2015 ). These words categories with examples are: Ability (brillian*, capab*, expert*, proficien*); Achievement (accomplish*, award*, power*, succeed*); Agentic (ambiti*, assert*, confident*, decisive*); Research (data, experiment*, manuscript*, research*); Standout adjectives (extraordinar*, remarkable, superb*, unique); Positive evaluation (appropriat*, clear*, innovat*, quality); and Negative evaluation (bias*, concern*, fail*, inaccura*). Higher Ability refers to higher usage of adjectives that describe talent, skill, or proficiency in a particular area; higher Achievement score refers to higher usage of terms that relate to success and achievement; higher Agentic score points to higher usage of words that describe achieving goals; higher Research score indicates higher usage of research terminology; higher Standout adjectives score reflects the use of adjectives describing exceptional, noticeable skill or performance; and higher Positive (Negative) evaluation score indicates higher (lower) display of affirmation and acceptance.

Word embeddings and t-distributed stochastic neighbour embedding

We also explored whether the texts of the articles and peer review reports from the two research disciplines formed different word clusters.

For peer review reports, we applied Word Embeddings, a method in which words are given mathematical vector representation so that they are mapped to points in Euclidean space, with words that are similar in meaning being closer to each other (Hren et al., 2022 ; Jurafsky et al., 2000 ). We used the Gensim library and Word2Vec approach in Python to create Word Embeddings, as well as a pre-trained model ( https://github.com/lintseju/word_embedding ), trained on Wikimedia database dump of the English Wikipedia on February 20, 2019. Finally, clusters were visualised using TensorBoard Embedding Projector ( https://projector.tensorflow.org/ ), which projects the high-dimensional data into three dimensions. The projector uses the principal components analysis (PCA) to visualise clusters and cosine distances between clusters as a reference to cluster distances. After the data points were created, we attached the labels by uploading a metadata file we previously generated with a Python script. We applied this technique only to peer review reports because the texts of the articles were a too large for TensorBoard Embedding Projector.

For article texts we applied t-Distributed Stochastic Neighbor Embedding (t-SNE) technique using the TSNE class from the Scikit-learn (Sklearn) Python library ( https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html ), which visualises high-dimensional data by placing each datapoint in a two-dimensional map (van den Maaten & Hinton, 2008 ). We used the Global Vectors for Word Representation (GloVe) pre-trained model that was trained using Wikipedia 2014 + Gigaword 5 (Pennington et al., 2014 ).

Word frequency in peer review reports

We also examined whether there would be differences in peer review reports in Social versus Medical and Health Sciences regarding the most common words usage. We calculated the percentage of words found in the 10,000 most common English words in the Project Gutenberg list (Wiktionary, 2006 ), as well as in the Academic Word List (AWL), which contains 570 words that are specific to written academic texts but are not included in the 2000 English words from the General service List (Coxhead, 2000 ; Coxhead & Nation, 2001 ). Finally, we identified the most frequent words that were unique to peer review reports in Social Sciences (i.e. not found in peer review reports in Medical Sciences in our sample) and vice versa.

Changes in manuscript versions: Levenshtein distance

Because of a small number of strictly Social Sciences articles, for this analysis we included Social Sciences articles that overlaped with Biology and Life Science, yielding 48 social and 166 articles from Medical and Health Sciences.

We measured the changes in the text from the first draft to the second version of a manuscript by means of the Levenshtein distance (Levenshtein, 1966 ), a character-based metric counting the minimum number of edit operations (insertions, deletions or substitutions) required to transform one text into the other. We computed this distance on the overall textual content of the manuscript, including figure captions and tables, but skipping references.

Changes in the references were calculated as the ratio of references being edited (added or deleted) to the total number of distinct references in both the first and the last version of the manuscript. To give two examples, we measured a change as 0.5 when the references of a manuscript changed from [A, B, C] to [A, B, D] (i.e. 2 changes of 4 references) and we calculated a value of 0.25 for a change from [A, B, C, D] to [A, B, C] (i.e. 1 change of 4 references). Two references were considered as equal by a matching algorithm (Vincent-Lamarre & Larivière, 2021 ) that checked whether they had the same publication year, the same number of authors, and a Levenshtein distance lower than 0.1 between the list of authors and the paper titles.

Statistical analysis

To assess possible differences between the articles and their reviews across Medical and Health, and Social Sciences, one-way ANOVA and post hoc Tukey test were employed. For multivariate frequency distribution of the variables, Contingency test was utilised. All analyses were carried out in JASP, Version 0.14.1. (JASP Team, 2020 ).

Structural and linguistic differences of articles in Social Sciences vs. Medical and Health Sciences

Articles from Medical and Health Sciences and Social Sciences differed in their structure (Table 1 ). Both the Introduction and Conclusion sections were longer for Social Sciences than those from Medical and Health Sciences. The number of additional sections and the number of special sections was also higher for articles in Social Sciences. Discussion and Conclusion sections were merged more often in Social Sciences articles, whereas Medical and Health Sciences articles often had Conclusions as a separate section. Articles in Medical and Health Sciences followed the IMRaD structure more frequently, and contained more figures.

Linguistic analysis was performed on the text of the latest version of an article. Articles in Social Sciences had higher Word count and higher Clout, whereas the articles in Medical and Health Sciences had a higher Tone score.

Characteristics of the peer review process and peer review reports from all stages of review in Social Sciences vs. Medical and Health Sciences

No statistically significant differences were found in the characteristics of the peer review reports from all stages of review between articles from Social Sciences and Medical and Health Sciences: reviewer’s gender, time between the first and second manuscript version posted, time between acceptance for publication and the first article version, time between the first and finally approved article version, as well as the number of article versions (Table 2 ).

Comparison of linguistic characteristics of peer review reports in Social Sciences vs. Medical and Health Sciences

Peer review reports were significantly longer for articles in Social Sciences (Table 3 ). Social Sciences peer review reports also had higher scores on several linguistic characteristics: Clout, Authentic, Tone, Agentic, Achievement, Research and Standout (Table 3 ).

Linguistic differences between research articles and corresponding peer reviews in Social Sciences vs. Medical and Health Sciences

We also compared the linguistic characteristics of the articles and their corresponding peer review reports (Table 4 ). In general, the articles had higher scores for the Analytic, Clout and Authentic variables than corresponding peer review reports, whereas peer review reports had a significantly higher positive Tone compared to the language of the corresponding articles.

The difference between the linguistic characteristics of articles and peer review reports was significantly higher for the Clout score for Social Sciences than for Medical and Health Sciences articles (Table 4 ). The opposite was true for the Tone score, where the difference between the articles and peer review reports was greater for Medical and Health Sciences articles (Table 4 ).

Comparison of reviewers’ recommendations for articles in Social Sciences vs. Medical and Health Sciences

There were no statistically significant differences in the outcome of the peer review process between the two disciplines, measured as the number of reviewers’ recommendations for different versions of the articles (Table 5 ). For articles in both disciplines, about a half of the articles were approved already at the stage of the first version. The proportion of reject recommendations were low and decreased in the next versions, with only a single article (in Medical and Health Sciences) receiving a rejection recommendation at the level of the third article version (Table 5 ).

Changes in the text of the manuscript were mainly concentrated in the Methods and Results & Discussion sections (Table 6 ), as measured by the higher values of the Levenshtein distance metric. References also changed, predominantly by adding more references. The differences between the disciplines were statistically significant only for the Introduction section, which was the least modified section in the Medical and Health Sciences.

Changes to the latest version of articles in Social Sciences vs. Medical and Health Sciences

Word cluster visualisation.

Using the Word Embedding visualisation of words, we observed that, the words in peer review reports from Social Sciences were more spherically distributed, which means that they had more general terms that could be found in other research areas, for example (Fig.  3 A). On the other hand, clusters consisting of specific terms were found in Medical and Health Sciences peer reviews (Fig.  3 B).

Word Embeddings 3D visualisation for reviews in Social Sciences (left) vs. Medical and Health Sciences (right). Clustering indicates grouping together the closest or most similar words; the closer two words are, the more similar they are, and vice versa. Gensim library and Word2Vec approach in Python were used to create Word Embeddings, as well as a pre-trained model ( https://github.com/lintseju/word_embedding ), trained on Wikimedia database dump of the English Wikipedia on February 20, 2019 and the clusters were visualised using TensorBoard Embedding Projector ( https://projector.tensorflow.org/ )

Using the t-Distributed Stochastic Neighbor Embedding (t-SNE) visualisation, we observed similar distributions for the words in text of the articles: words from Social Sciences were more spherically distributed as well, having more general terms (Fig.  4 A) while clusters consisting of specific terms were found in Medical articles texts (Fig.  4 B). GIF format of the images can be found in the Appendix.

The-Distributed Stochastic Neighbor Embedding (t-SNE) visualisation (van den Maaten & Hinton, 2008 ) for texts of the articles in Social Sciences ( A ) vs. Medical and Health Sciences ( B ) in a two-dimensional map. Created using the TSNE class from the Scikit-learn (Sklearn) Python library (Skicit learn, 2022 ) and the Global Vectors for Word Representation (GloVe) pre-trained model that was trained using Wikipedia 2014 + Gigaword 5 (Pennington et al., 2014 )

Peer review reports from Social Sciences contained a higher percentage of words from the Academic Word List (8.5%) compared to peer review reports from Medical and Health Sciences (7.2%) (MD = − 1.3, 95% CI − 1.7 to − 0.8). They also contained a higher percentage of the 10,000 most common English words (76.7%) compared to peer review reports from Medical and Health Sciences (72.7%) (MD = − 4.0, 95% CI − 5.0 to − 3.0).

Most common and most common unique words found in peer reviews can be found in Table 7 .

Understanding the differences between Medical and Health Sciences and Social Sciences in their structural and linguistic characteristics is crucial for successful interdisciplinary collaborations and for avoiding misunderstandings between different research groups. Our study found certain differences both in articles and peer review reports regarding to their structure and linguistic characteristics.

Structural differences in articles and peer review reports between Social and Medical and Health Sciences

Longer articles and peer review reports in Social Sciences compared to Medical and Health Sciences could reflect the tradition of the writing style and formats typically used in the disciplines. Despite an increase of journal articles as a publication output for social sciences and humanities (Savage & Olejniczak, 2022 ), academic monographs and books are still being used as forms of scholarly dissemination in the humanities and some social sciences, sometimes even remaining crucial for professional advancement (Williams et al., 2009 ). Some university departments emphasise publishing in the form of books and monographs (Wolfe, 1990 ). Typically, monographs range between 70,000 and 110,000 words, which makes them significantly longer than a standard or even the longest journal article. The length of journal articles also differs across disciplines, with medical journals usually strict limits for article word count. For example, in five medical journals ( New England Journal of Medicine [ NEJM ], Lancet , JAMA , BMJ and Annals of Internal Medicine ), the word limits for the main text ranges from 2700 ( NEJM ) to 4400 ( BMJ ) (Silverberg & Ray, 2018 ). On the other hand, social sciences journals allow longer papers and they typically limit the manuscript size in the number of pages. For example, in four social sciences journals ( Review of Economics and Statistics [ Rev Econ Stat ], Journal of Business and Economic Statistics [ JBES ], Human relations , Journal of Marriage and Family [ JMF ]), page limits for the main text ranged from 35 ( JBES and JMF ) to 45 ( Rev Econ Stat ), the limit often being a recommendation rather than obligation. Some journals, such as Sociological Science , do not have any limits related to manuscript length. A 2011 market research conducted by Palgrave Macmillan, a publisher of books and journals in humanities and social sciences, in which they surveyed 1,268 authors and academics from humanities and social sciences, the majority of respondents expressed that the perfect length would be between a journal article and a monograph (McCall, 2015 ). This resulted in the development of Palgrave Pilot, a format ranging between 25,000 and 50,000 words (McCall, 2015 ).

Medical and Health Science articles were shorter, but contained more images and graphs compared to Social Sciences articles. However, studies suggest that even in medical sciences journals, graphs are underused (Chen et al., 2017 ), they are often not self-explanatory and fail to display full data (Cooper et al., 2001 ). Peer review seems to improve graph quality but there is further need for improvement (Schriger et al., 2016 ). Because social sciences are entering a golden age (Salganik, 2019 ), with more data available (Buyalskaya et al., 2021 ), social sciences authors should also recognize the importance of the visual data presentation and increase the number of graphs and figures.

As expected, Medical and Health Science articles more often followed the IMRaD format compared to Social Sciences, since they were among the first to adopt IMRaD structure. This is not surprising since research in health and life sciences most often uses a hypothetico-deductive approach (Jürgen, 1968 ; Lewis, 1988 ), which starts from a hypothesis, moves to observation and comes to a conclusion. IMRaD is the perfect format to present such research as it follows the structure of a logical argument (Puzzo & Conti, 2021 ). social sciences, on the other hand, widely use a mixed method approach (Plano Clark & Ivankova, 2016 ; Timans et al., 2019 ), which incorporates both deductive and inductive methods (Creswell, 2012 ). Inductive approach moves from observation to hypothesis, and the IMRaD format may not be suitable. As there is an increase in mixed method approach in health and clinical sciences (Plano Clark, 2010 ; Coyle et al., 2018 ), it is questionable whether IMRaD can and should be a one-size-fits-all format of journal article. If research is done inductively, should it be presented in IMRaD format? There are even arguments that the current publishing process discourages inductive research (Woiceshyn & Daellenbach, 2018 ). Nevertheless, as the format of research paper has evolved from descriptive to standardised style (Kronick, 1976 ), IMRaD format will continue to evolve as well (Wu, 2011 ) to adapt to the diversification of methodological approaches in different scientific disciplines and particularly in multi- and interdisciplinary work.

Linguistic differences in articles and peer review reports between Social Sciences and Medical and Health Sciences

Articles in Social Sciences had higher Word count, Clout, and Authenticity, whereas articles in Medical and Health Sciences had higher Analytic and Tone score. Higher Authenticity score for social sciences articles, which indicates a more personal way of writing, is not an unexpected finding as analyses in social sciences and humanities often relay on interpretations based on researcher’s personal opinions and values, leading to subjectivity (Khatwani & Panhwar, 2019 ). Higher Analytic score for articles in Medical and Health Sciences, which reflects the use of formal, logical, and hierarchical language, does not surprise due to the hypothetico-didactic methodological approach and “dispassionate scientific language” that is frequently used in these disciplines (Steffens, 2021 ). This is partially in accordance with previous studies that compared peer review reports in social sciences and medical and health sciences. Understanding the language differences between disciplines is important because linguistic characteristics of manuscripts may have an effect on the evaluation process. Peer review has a crucial role in determining the fate of manuscripts. If peer review reports contain more positive words and/or expressions, research manuscripts are more likely to be accepted for publishing (Fadi Al-Khasawneh, 2022 ; Ghosal et al., 2019 ; Wang & Wan, 2018 ). Also, the absence of negative comments can indicate a positive outcome for the submitted article (Thelwall et al., 2020 ). There is a positive correlation between longer texts and longer sentences, and the positive score of the selection procedures (van den Besselaar & Mom, 2022 ). Furthermore, project descriptions with a more pronounced narrative structure and expressed self-confidence are more likely to be granted (van den Besselaar & Mom, 2022 ).

We also found linguistic differences in the peer review reports between the two research areas. Peer review reports for Social Sciences articles had higher scores on several linguistic characteristics: Clout, Authentic, Tone, Agentic, Achievement, Research and Standout. On the other hand, peer review reports for Medical and Health Sciences had higher score on positive evaluation words, i.e. more positive descriptors and superlatives, than the reports in Social Sciences. This is partially in accordance with previous studies that compared peer review reports in social and medical and health sciences. Buljan et al. ( 2020 ) found that the language of peer review reports in social sciences journals had high Authenticity and Clout scores, whereas peer review reports in medicine had higher Analytical tone than peer review reports in social sciences. In addition, reviewer recommendations were closely associated with the linguistic characteristics of the review reports, and not to area of research, type of peer review, or reviewer gender (Buljan et al., 2020 ). Our study, on the other hand, showed that there were differences of the linguistic characteristics of articles and peer review reports between Social Sciences and Medical and Health Sciences. As ORC contains only open peer review reports, the question remains whether there would be differences between open and closed peer review reports. For example, a study showed that closed peer review reports had more positive LIWC Tone compared to open peer review (Bornmann et al., 2012 ).

One of the novelties that our study brings is the comparison of words used in peer review reports in Social Sciences and Medical and Health Sciences. While Social Sciences peer review reports had more general terms that could be found in other research areas, terminology in Medical and Health Sciences peer review reports was more profession-specific. We visualised these results using clusters to indicate grouping together the closest or most similar words: the closer two words are, the more similar they are, and vice versa. More clusters consisting of specific terms were found in peer review reports in Medical and Health Sciences than in the Social Sciences. This finding actually confirms medical terminology as one of the “oldest specialized terminologies in the world”, having been shaped from Greek and Latin medical writings for over 2000 years (Džuganová, 2019 ).

Characteristics of the peer review process

We found no statistically significant differences in the duration of the peer review process between Social Sciences and Medical and Health Sciences for articles published in post-publication peer-review platforms. About a half of the articles in both disciplines were approved at the stage of the first version. The reason for this is probably the uniform policy and procedures of the Open Research Central platform, i.e. the same evaluation process for articles in both disciplines. The duration of the peer review process may differ across research areas. A study on 3500 peer review experiences published at the SciRev.sc website revealed significant differences in the duration of the first round and of the total review process across research areas. The first round was of the shortest duration in medicine and public health journals, lasting 8–9 weeks while it was twice as longer in social sciences and humanities, approximately 16–18 weeks (Huisman & Smits, 2017 ). The study also showed that the total peer review duration in medicine and public health journals was 12–14 weeks, whereas in social sciences and humanities journals about 22–23 weeks (Huisman & Smits, 2017 ). We also did not find statistically significant differences in other characteristics of the peer review process between the two research areas, such as reviewer’s gender or the number of article versions.

Changes in the manuscript versions

We found that differences between research areas were only statistically significant in the Introduction section, which was the least modified section in the Medical and Health Sciences manuscripts. Some studies examined whether the manuscript versions changed based on the peer review reports. Nicholson et al. ( 2022 ) compared linguistic features within bioRxiv preprints to published biomedical texts with aim of examining their changes after peer review. Among predominant changes were typesetting, mentions of supporting information sections or additional files. Another study (Akbaritabar et al., 2022 ) matched 6024 preprint-publication pairs across research areas and examined changes in their reference lists between the manuscript versions. They found that 90% of references were not changed between versions and 8% were added. The study also found that manuscripts in the natural and medical sciences reframe their literature more extensive, whereas changes in engineering were mostly related to methodology.

Limitations

The limitation of our study is that the Social Sciences articles and peer review reports that we used were mostly from Psychology and Sociology, which have structural similarities to those from Medical and Health Sciences. For example, articles from these two disciplines tend to have IMRaD structure, similar to articles from Medical and Health Sciences. The limitation is also the difference in the sample size as the platform journals still predominantly publish Medical and Health research.

Recommendations

Are the similarities we found between articles in Social Sciences and Medical and Health Sciences a result of their real differences or because the authors had to use the same format of the ORC platforms? The same question applies for the peer review reports. We believe this is due to the latter. For this reason, we recommend that the editors of all ORC platforms take potential structural and linguistic differences between disciplines in consideration. We believe the editors should also consider whether the IMRaD structure is the most appropriate format for each of the disciplines and whether additional formats should be offered.

Due to the different approach, tradition of the writing style and formats typically used in the two compared disciplines, it is not surprising that there are structural and linguistic differences in research articles in Medical and Health Sciences and Social Sciences. However, the review process for articles in Social Sciences and Medical and Health Sciences may not differ as much as is usually considered. This may be due in part to the same platform, which may have uniform policies and processes. With the development of open science practices in social sciences (Christensen et al., 2019 ), publishing platforms from social sciences and humanities that offer open peer review (Palgrave MacMillan, 2014 ), those that host multidisciplinary journals (Tracz, 2017 ; ORC, 2022 ), and with the evolving role of preprints (Mirowski, 2018 ) and editorial and review innovations, we can perhaps expect even greater conversion of the article formats and evaluation processes across research areas.

Akbaritabar, A., Stephen, D., & Squazzoni, F. (2022). A study of referencing changes in preprint-publication pairs across multiple fields. Journal of Informetrics, 16 (2), 101258. https://doi.org/10.1016/j.joi.2022.101258

Article   Google Scholar  

Al-Khasawneh, F. (2022). Analysis of the language used in the reports of peer-review journals. Applied Research on English Language, 11 , 79–94. https://doi.org/10.22108/are.2022.130458.1774

Bornmann, L., Wolf, M., & Daniel, H.-D. (2012). Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics, 91 , 843–856. https://doi.org/10.1007/s11192-011-0569-5

Buljan, I., Garcia-Costa, D., Grimaldo, F., Squazzoni, F., & Marušić, A. (2020). Large-scale language analysis of peer review reports. eLife, 17 (9), e53249. https://doi.org/10.7554/eLife.53249

Buyalskaya, A., Gallo, M., & Camerer, C. F. (2021). The golden age of social science. PNAS, 118 (5), e2002923118. https://doi.org/10.1073/pnas.2002923118

Chen, J. C., Cooper, R. C., McMullen, M. E., & Schriger, D. L. (2017). Graph quality in top medical journals. Annals of Emergency Medicine, 69 (4), 453–461. https://doi.org/10.1016/j.annemergmed.2016.08.463

Christensen, G., Wang, Z., Levy Paluck, E., Swanson, N., Birke, D., J., Miguel, E., & Littman, E. (2019). Open science practices are on the rise: The state of social science (3S) survey. MetaArxiv Preprints. Preprint. August 2019. Retrieved from https://osf.io/preprints/metaarxiv/5rksu/ .

Cooper, R. J., Schriger, D. L., & Close, R. J. H. (2001). Graphical literacy: The quality of graphs in a large-circulation journal. Annals of Emergency Medicine, 40 (3), 317–322. https://doi.org/10.1067/mem.2002.127327

Coxhead, A. (2000). A new academic word list. TESOL Quarterly, 34 (2), 213–238. https://doi.org/10.2307/3587951

Coxhead, A., & Nation, P. (2001). The specialised vocabulary of english for academic purposes. In J. Flowerdew & M. Peacock (Eds.), Research perspectives on english for academic purposes. Cambridge University Press.

Google Scholar  

Coyle, C. E., Schulman-Green D., Feder, S., Toraman, S., Prust, M. L., & Plano Clark, V. L. (2018). Federal Funding for Mixed Methods Research in the Health Sciences in the United States: Recent Trends. Journal of Mixed Methods Research, 12 (3), 305–324. https://doi.org/10.1177/1558689816662578

Creswell, J. (2012). Qualitative inquiry and research design: Choosing among five approaches . Sage.

Džuganová, B. (2019). Medical language – a unique linguistic phenomenon. JAHR - European Journal of Bioethics, 10 (1), 129–145. https://doi.org/10.21860/j.10.1.7

Ford, E. (2015). Open peer review at four STEM journals: an observational overview. F1000Research. https://doi.org/10.12688/f1000research.6005.2

Garcia-Costa, D., Squazzoni, F., Mehmani, B., & Grimaldo, F. (2022). Measuring the developmental function of peer review: A multi-dimensional, cross-disciplinary analysis of review reports from 740 academic journals. PeerJ, 10 , e13539. https://doi.org/10.7717/peerj.13539

Ghosal, T., Verma, R., Ekbal, A., & Bhattacharyya, P. (2019). DeepSentiPeer: Harnessing sentiment in review texts to recommend peer review decisions. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, presented at the ACL 2019, Association for Computational Linguistics, Florence, Italy, pp. 1120–1130.

Hamilton, D. G., Fraser, H., Hoekstra, R., & Fidler, F. (2020). Journal policies and editors’ opinions on peer review. eLife, 9 , e62529. https://doi.org/10.7554/eLife.62529

Hren, D., Pina, D. G., Norman, C. R., & Marusic, A. (2022). What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports. Journal of Informetrics, 16 (2), 101289. https://doi.org/10.1016/j.joi.2022.101289

Huisman, J., & Smits, J. (2017). Duration and quality of the peer review process: The author’s perspective. Scientometrics, 113 (1), 633–650. https://doi.org/10.1007/s11192-017-2310-5

Jaffe, K. (2014). Social and natural sciences differ in their research strategies, adapted to work for different knowledge landscapes. PLoS One, 9 (11), e113901. https://doi.org/10.1371/journal.pone.0113901

JASP Team (2020). JASP (Version 0.14.1) [Computer software]. Retrieved from https://jasp-stats.org/ .

Jurafsky, D., & Martin, H.J. (2000). Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition. Upper Saddle River, N.J.: Prentice Hall. ISBN 978-0-13-095069-7. Retrieved from https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf .

Jürgen, H. (1968). Erkenntnis und Interesse. Suhrkamp, Frankfurt am Main 1968

Kaatz, A., Magua, W., Zimmerman, D. R., & Carnes, M. (2015). A quantitative linguistic analysis of national institutes of health R01 application critiques from investigators at one institution. Academic Medicine, 90 (1), 69–75. https://doi.org/10.1097/ACM.0000000000000442

Karhulahti, V. M., & Backe, H. J. (2021). Transparency of peer review: A semi-structured interview study with chief editors from social sciences and humanities. Research Integrity and Peer Review, 6 (1), 13. https://doi.org/10.1186/s41073-021-00116-4

Khatwani, M. K., & Panhwar, F. Y. (2019). Objectivity in social research: A critical analysis. Asia Pacific, 37 , 126–142.

Kronick, D. A. (1976). A history of scientific & technical periodicals: The origins and development of the scientific and technical press (pp. 1665–1790). Scarecrow Press.

Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10 (8), 707–710.

MathSciNet   Google Scholar  

Lewis, R. W. (1988). Biology: A hypothetico-deductive science. The American Biology Teacher, 50 (6), 362–366.

McCall, J. (2015). Format, flexibility, and speed. The Academic Book of the Future . Retrieved from https://academicbookfuture.org/2015/06/12/format-flexibility-and-speed-palgrave-pivot/ .

Mirowski, P. (2018). The future(s) of open science. Social Studies of Science, 48 (2), 171–203. https://doi.org/10.1177/0306312718772086

Ngai, S. B. K., Gill Singh, R., & Chun Koon, A. (2018). A discourse analysis of the macro-structure, metadiscoursal and microdiscoursal features in the abstracts of research articles across multiple science disciplines. PLoS One, 13 (10), e0205417. https://doi.org/10.1371/journal.pone.0205417

Nicholson, D. N., Rubinetti, V., Hu, D., Thielk, M., Hunter, L. E., & Greene, C. S. (2022). Examining linguistic shifts between preprints and publications. PLoS Biology, 20 (2), e3001470. https://doi.org/10.1371/journal.pbio.3001470

Open Research Central (ORC). (2022). Retrieved from https://openresearchcentral.org/

Palgrave Macmillan. (2014). Retrieved from https://palgraveopenreview.wordpress.com/

Pennebaker, J. W., Booth, R. J., Boyd, R. L., & Francis, M. E. (2015). Linguistic Inquiry and Word Count: LIWC2015a . Pennebaker Conglomerates.

Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015b). The development and psychometric properties of LIWC2015 . University of Texas at Austin.

Pennington, J., Socher, R., & Manning, C.D. (2014). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1 October 2014. https://doi.org/10.3115/v1/D14-1162

Plano Clark, V., & Ivankova, N. (2016). Mixed methods research: A guide to the field . SAGE Publications.

Book   Google Scholar  

Plano Clark, V. L. (2010). The Adoption and Practice of Mixed Methods: U.S. Trends in Federally Funded Health-Related Research. Qualitative Inquiry, 16 (6), 428–440. https://doi.org/10.1177/1077800410364609

Puzzo, D., & Conti, F. (2021). Conceptual and methodological pitfalls in experimental studies: An overview, and the case of alzheimer’s disease. Frontiers in Molecular Neuroscience, 15 (14), 684977. https://doi.org/10.3389/fnmol.2021.684977

Rashidi, K., Sotudeh, H., Mirzabeigi, M., & Nikseresht, A. (2020). Determining the informativeness of comments: A natural language study of F1000Research open peer review reports. Online Information Review, 44 (7), 1327–1345. https://doi.org/10.1108/OIR-02-2020-0073

Ross-Hellauer, T. (2017). What is open peer review? A systematic review. F1000Research, 6 , 558. https://doi.org/10.12688/f1000research.11369.2

Salganik, M.J. (2019). Bit by Bit: Social research in the digital age. Princeton University Press, SBN: 9780691196107.

Savage, W. E., & Olejniczak, A. J. (2022). More journal articles and fewer books: Publication practices in the social sciences in the 2010’s. PLoS One, 17 (2), e0263410. https://doi.org/10.1371/journal.pone.0263410

Schmader, T., Whitehead, J., & Wysocki, V. H. (2007). A linguistic comparison of letters of recommendation for male and female chemistry and biochemistry job applicants. Sex Roles, 57 (7–8), 509–514. https://doi.org/10.1007/s11199-007-9291-4

Schriger, D. L., Raffetto, B., Drolen, C., & Cooper, R. J. (2017). The Effect of Peer Review on the Quality of Data Graphs in Annals of Emergency Medicine. Annals of Emergency Medicine, 69 (4), 444–452. https://doi.org/10.1016/j.annemergmed.2016.06.046

Silverberg, O., & Ray, J. G. (2018). Variations in instructed vs. published word counts in top five medical journals. Journal of General Internal Medicine, 33 (1), 16–18. https://doi.org/10.1007/s11606-017-4196-6

Skicit learn 2022. Retrieved from https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html

Sollaci, L. B., & Pereira, M. G. (2004). The introduction, methods, results, and discussion (IMRAD) structure: A fifty-year survey. Journal of the Medical Library Association, 92 (3), 364–371.

Squazzoni, F., Ahrweiler, P., Barros, T., Bianchi, F., Birukou, A., Blom, H., Bravo, G., Cowley, S., Dignum, V., Dondio, P., Grimaldo, F., Haire, L., Hoyt, J., Hurst, P., Lammey, R., MacCallum, C., Marušić, A., Mehmani, B., Murray, H., Nicholas, D., … Willis, M. (2020). Unlock ways to share data on peer review. Nature, 578 (7796), 512–514. https://doi.org/10.1038/d41586-020-00500-y

Squazzoni, F., Bravo, G., Farjam, M., Marusic, A., Mehmani, B., Willis, M., Birukou, A., Dondio, P., & Grimaldo, F. (2021). Peer review and gender bias: A study on 145 scholarly journals. Science Advances . https://doi.org/10.1126/sciadv.abd0299

Squazzoni, F., Bravo, G., Grimaldo, F., García-Costa, D., Farjam, M., & Mehmani, B. (2021). Gender gap in journal submissions and peer review during the first wave of the COVID-19 pandemic. A study on 2329 Elsevier journals. PLoS One, 16 (10), e0257919. https://doi.org/10.1371/journal.pone.0257919

Steffens, A. N. V., Langerhuizen, D. G. W., Doornberg, J. N., Ring, D., & Janssen, S. J. (2021). Emotional tones in scientific writing: Comparison of commercially funded studies and non-commercially funded orthopedic studies. Acta Orthopaedica, 92 (2), 240–243. https://doi.org/10.1080/17453674.2020.1853341

Thelwall, M., Papas, E.-R., Nyakoojo, Z., Allen, L., & Weigert, V. (2020). Automatically detecting open academic review praise and criticism. Online Information Review, 44 (5), 1057–1076. https://doi.org/10.1108/OIR-11-2019-0347

Timans, R., Wouters, P., & Heilbron, J. (2019). Mixed methods research: What it is and what it could be. Theory and Society, 48 , 193–216. https://doi.org/10.1007/s11186-019-09345-5

Tracz, V. (2017). ORC – Open Research Central: ‘repulsive and malevolent’ or ‘lover of rebellion and freedom’. F1000 blognetwork. Retrieved from https://blog.f1000.com/2017/07/12/orc-open-research-central-repulsive-and-malevolent-or-lover-of-rebellion-and-freedom/ .

Trix, F., & Psenka, C. (2003). Exploring the color of glass: letters of recommendation for female and male medical faculty. Discourse & Society , 14(2), 191–220.

van den Besselaar, P., & Mom, C. (2022). The effect of writing style on success in grant applications. Journal of Infometrics, 16 , 101257. https://doi.org/10.1016/j.joi.2022.101257

van der Maaten, L. J. P., & Hinton, G. E. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9 , 2579–2605.

MATH   Google Scholar  

Vincent-Lamarre, P., & Larivière, V. (2021). Textual analysis of artificial intelligence manuscripts reveals features associated with peer review outcome. Quantitative Science Studies, 2 (2), 662–677. https://doi.org/10.1162/qss_a_00125

Wang, K. & Wan, X. (2018). Sentiment analysis of peer review texts for scholarly papers, The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA, pp. 175–184.

Wiktionary, the free dictionary: Frequency lists. (2006). Retrieved from https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/PG/2006/04/1-10000 .

Williams, P., Stevenson, I., Nicholas, D., Watkinson, A., & Rowlands, I. (2009). The role and future of the monograph in arts and humanities research. Aslib Proceedings, 61 (1), 67–82. https://doi.org/10.1108/00012530910932294

Woiceshyn, J., & Daellenbach, U. (2018). Evaluating inductive vs deductive research in management studies: Implications for authors, editors, and reviewers. Qualitative Research in Organizations and Management, 13 (2), 183–195. https://doi.org/10.1108/QROM-06-2017-1538

Wolfe, A. (1990). Books vs. articles: Two ways of publishing sociology. Sociological Forum, 5 (3), 477–489.

Wu, J. (2011). Improving the writing of research papers: IMRAD and beyond. Landscape Ecology, 26 , 1345–1349. https://doi.org/10.1007/s10980-011-9674-3

Download references

This research received funding from the Croatian Science Foundation within the framework “Professionalism in Health: decision-making in health and research”—Pro Dem under Grant Agreement No [IP-2019-04-4882].

Author information

Andrijana Perković Paloš and Antonija Mijatović shared first authorship.

Authors and Affiliations

School of Medicine, Center for Evidence-Based Research and Department of Research in Biomedicine and Health, University of Split, Šoltanska 2, Split, Croatia

Andrijana Perković Paloš, Antonija Mijatović, Ivan Buljan & Ana Marušić

Department of Psychology, Faculty of Humanities and Social Sciences, University of Split, Split, Croatia

Ivan Buljan

Department of Computer Science, University of Valencia, Valencia, Spain

Daniel Garcia-Costa, Elena Álvarez-García & Francisco Grimaldo

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Andrijana Perković Paloš or Antonija Mijatović .

Ethics declarations

Competing interests.

The Authors confirm that there are no relevant financial or non-financial competing interests to report.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Perković Paloš, A., Mijatović, A., Buljan, I. et al. Linguistic and semantic characteristics of articles and peer review reports in Social Sciences and Medical and Health Sciences: analysis of articles published in Open Research Central. Scientometrics 128 , 4707–4729 (2023). https://doi.org/10.1007/s11192-023-04771-w

Download citation

Received : 02 December 2022

Accepted : 07 June 2023

Published : 03 July 2023

Issue Date : August 2023

DOI : https://doi.org/10.1007/s11192-023-04771-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Peer review
• Social Sciences
• Medical and Health Sciences
• Research article
• Find a journal
• Publish with us
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 23 April 2024

Research on flipped classrooms in foreign language teaching in Chinese higher education

• Wen Kong 1 ,
• Di Li 2 &
• Quanjiang Guo   ORCID: orcid.org/0000-0002-7846-1363 3  

Humanities and Social Sciences Communications volume  11 , Article number:  525 ( 2024 ) Cite this article

182 Accesses

Metrics details

• Language and linguistics

This review examines 233 articles published in Chinese academic journals between 2011 and 2021, documenting the state of research concerning flipped classrooms (FCs) in foreign language teaching within the context of higher education in China. Employing the methodological approach of a scoping review, the investigation is underpinned by the five-stage framework articulated by Arksey and O’Malley. The results reveal a notable surge in FC-related studies between 2013 and 2017, with a subsequent decline in scholarly attention. The majority of the reviewed studies on FCs focused on English instruction at the college level, with a conspicuous dearth of inquiry into the application of FCs in the teaching of other foreign languages. All studies were categorized as either empirical or non-empirical, and the most frequently used instruments for data collection were surveys and interviews; case studies were underrepresented in the literature. Early studies focused on the introduction of the new model, while more recent investigations focused on the impact of its implementation. The findings of the in-depth content analysis unearthed a prevailing trend of high learner satisfaction with the FC model, along with favorable direct and indirect educational outcomes. Noteworthy factors influencing the efficacy of FCs included learners’ foreign language proficiency and their self-regulation or self-discipline abilities. The paper concludes with a discussion of the challenges in FC implementation and a call for future research on this promising pedagogy.

Similar content being viewed by others

The implementation of a flipped classroom approach at a UK dental school

Effectiveness of the flipped classroom model on students’ self-reported motivation and learning during the COVID-19 pandemic

Content validity of the Constructivist Learning in Higher Education Settings (CLHES) scale in the context of the flipped classroom in higher education

Introduction.

The flipped classroom (FC), also known as the “inverted classroom”, is a pedagogical approach that first emerged in the 1980s and came into more widespread use in the 2000s (Baker, 2000 ; Bergmann and Sams, 2012 ; Khan, 2012 ). It has gained prominence as advances in technology afford increasing opportunities for ubiquitous access to a variety of online resources. The FC model removes in-class lectures, freeing up classroom time for more in-depth exploration of topics through discussion with peers or problem-solving activities facilitated by instructors. The removed content is often delivered to learners through pre-class materials like video recordings. As a result, in the FC, learning activities that are active and social occur inside the classroom while most information transmission occurs outside the classroom. Today, the FC has been implemented in many different disciplines and in schools and universities around the world (Akcayir and Akcayir, 2018 ).

Proponents of the FC assert its pedagogical merits on several fronts. First, it alleviates the constraints associated with requiring all learning to happen at the same time and place, furnishing learners with an individualized education that enables flexible online study at their own pace as long as an internet connection is available (Hung, 2014 ). Second, it allocates class time to the cultivation of learners’ higher-order cognitive skills, emphasizing application, analysis, and evaluation, as opposed to lower-order skills such as knowledge and comprehension (Brinks-Lockwood, 2014 ; Lee and Wallace, 2018 ). Third, in contrast to traditional lecturing, the FC is a student-centered approach emphasizing engagement and active learning (Steen-Utheim and Foldnes, 2018 ), fostering students’ autonomy by endowing them with heightened responsibility for their learning (Brinks-Lockwood, 2014 ; O’Flaherty and Philips, 2015 ).

Vygotsky’s social constructivism ( 1978 ) has frequently been adopted as a theoretical foundation for designing learning experiences in technologically rich environments (Marzouki et al., 2017 ), and this framework highlights the particular benefits of technology-enhanced FC pedagogy (Jarvis et al., 2014 ). As mentioned above, in an FC model, learners can watch pre-recorded videos in their own time before class to remember basic information and understand concepts as they prepare for classroom activities, while the higher-order skills of analyzing, applying, evaluating, and creating can be collaborative and interactive, taking place in class with the guidance of a teacher, and thus facilitating progression within the learners’ proximal developmental zone.

Since its introduction in foreign language teaching (FLT) in China in 2011, the FC has attracted increasing research attention and has been welcomed by foreign language teachers (Yan and Zhou, 2021 ). Over the past decade, the Ministry of Education of the People’s Republic of China has exerted increasing pressure on higher education institutions to transition from traditional teacher-centered lecture-style approaches to innovative methods integrating technology and the internet, with the goals of enhancing learning, sustaining student engagement, and improving student satisfaction (Ministry of Education of People’s Republic China, 2021 ). The FC model, combined with traditional face-to-face teaching and personalized online learning, has emerged as a popular strategy in China to meet ministry requirements while delivering cost-effective and learner-centered curricula in response to the increasing student enrollment in higher education.

Despite the wide adoption of FCs in FLT in China, literature reviews about their implementation and effects have been notably scarce in the last decade. A search of the China National Knowledge Infrastructure (CNKI), the largest national research and information publishing company housing China’s most extensive academic database, revealed only three reviews—by Deng ( 2016 ), Qu ( 2019 ), and Su et al. ( 2019 )—published prior to the end of 2021. These reviews primarily focused on FCs in the context of English as a foreign language (EFL) education, overlooking most of the over 100 foreign languages taught in Chinese higher education. As a result, these reviews fell short of delivering a comprehensive analysis of research pertaining to FCs, and the reliability and generalizability of their findings in non-EFL contexts are questionable. Moreover, Deng ( 2016 ) and Su et al.’s (2019) reviews included all published papers without establishing clear inclusion and exclusion criteria. For example, they did not exclude articles that made a passing or token reference to the FC model, short papers of only one or two pages in length, book reviews, or editorials. Qu’s study ( 2019 ), on the other hand, was constrained in scope to articles within the Chinese Social Sciences Citation Index (CSSCI), a sub-database developed by Nanjing University of China Academy of Social Sciences Research Evaluation Center and the Hong Kong University of Science and Technology, and thus omitted relevant contributions from other academic journals. The CNKI incorporates both the CSSCI and the Core Journals of China (CJC), an equally significant sub-database overseen by the Peking University Library and experts from relevant institutions. Given the exclusion of the latter, a reevaluation of the scope and potential limitations of Qu’s study is warranted.

Thus, there persists an imperative for a comprehensive synthesis of the extant studies on FCs in FLT within Chinese higher education over the past decade. The restricted visibility of studies conducted in China, owing to their publication in Chinese and confinement to Chinese academic journals, makes it difficult for international researchers and practitioners to access and comprehend this body of literature. Such understanding among the global academic community is necessary for exploring both the strengths and limitations of FCs in diverse cultural and linguistic contexts.

Research method

The current study adopts a scoping review approach based on the methodological framework developed by Arksey and O’Malley ( 2005 ) to provide both quantitative and qualitative data for researchers and practitioners.

A scoping review is a relatively new approach to synthesizing research data which has been gaining popularity in many disciplines (Davis et al., 2009 ; Daudt et al., 2013 ). It is often undertaken as an independent project when a research area is complex, and no review of that area has previously been made available. A scoping review serves to highlight the relevant literature to researchers with the aim of rapidly mapping the key concepts characterizing a research area and the main sources and types of evidence available (Arksey and O’Malley, 2005 ; Mays et al., 2005 ; Levac et al., 2010 ). According to Arksey and O’Malley ( 2005 ), this kind of review addresses four goals: to examine the extent, range, and nature of research activity; to determine the value of undertaking a full systematic review; to summarize and disseminate research findings; and to identify research gaps in the existing literature. The scoping review is increasingly being employed in the field of foreign language education to provide a comprehensive view of FLT studies, identify implications for theory and pedagogy, or inform subsequent in-depth reviews and empirical studies (Chan et al., 2022 ; Hillman et al., 2020 ; Tullock and Ortega, 2017 ).

The difference between a scoping review and a narrative or traditional literature review lies in the transparency of the review process. A narrative review usually depends on the author’s own knowledge or experience to describe the studies reviewed and uses an implicit process to provide evidence (Garg et al., 2008 ). The reader cannot determine how much literature has been consulted or whether certain studies have been ignored due to contradictory findings. A scoping review, in contrast, uses an explicit, rigorous, and systematic approach to retrieve relevant articles to ensure the transparency and replicability of the data extraction process. For example, the methodological framework adopted by Arksey and O’Malley ( 2005 ) for conducting a scoping study comprises five stages: identifying the research questions; identifying relevant studies; selecting studies for inclusion; charting the data; and collating, summarizing, and reporting the results. By presenting the process and results in an accessible and summarized format, reviewers are in a position to illustrate the field of interest in terms of the volume, nature, and characteristics of the primary research, enabling researchers, practitioners, and policymakers to make effective use of the findings.

Figure 1 presents the process of the scoping review in the current study based on the five-stage methodological framework developed by Arksey and O’Malley ( 2005 ).

Process of the scoping review.

Process of the scoping review

Identifying research questions.

This scoping review is driven by four research questions:

RQ1. What is the current state of FC research in FLT within the context of higher education in China?

RQ2. What research methods and instruments have been employed in the included FC studies?

RQ3. What research foci and trends are displayed in the included FC studies?

RQ4. What are the major findings of the included FC studies?

RQ1 aims to provide an overview of studies on FCs in FLT in Chinese higher education by providing details about the basic information about existing publications, such as the number of publications per year and the distribution of publications by foreign language context. RQ2 leads to a classification of the research methods and instruments used to collect data in FC research. RQ3 explores the topics and trends in FC research over the past decade with the help of the literature visualization and analysis tool CiteSpace5.8R3. RQ4 reveals the effects of the FC model on direct and indirect educational outcomes, learners’ satisfaction with FCs, and the factors influencing the impact of FCs, as documented in the reviewed sources.

Searching for relevant studies

To be as comprehensive as possible in identifying primary evidence and to ensure the quality of the published articles, we searched both CSSCI and CJC in the CNKI database. The key search terms were developed and categorized based on two dimensions according to the purpose of the review. One dimension related to teaching or learning in FCs, while the other dimension related to the types of foreign languages. The key search terms and search methods are listed in Table 1 .

As the FC approach was introduced into FLT in China in 2011, the search included articles published between 2011 and 2021. Further inclusion and exclusion criteria were developed to focus on the scope of the review; these are outlined in Table 2 .

Study selection

Figure 2 shows a process diagram of the study selection process, which consisted of four phases: searching the databases; identifying the total number of articles in each database; screening titles, abstracts, and full texts; and selecting eligible articles for inclusion.

Flowchart diagram for article selection.

The final database search was conducted on January 16, 2022, and resulted in the identification of a total of 333 articles. Subsequently, all potentially relevant articles went through a three-step screening process. The first step excluded 9 duplicates. The second step excluded irrelevant articles by screening titles and abstracts; 37 articles were removed at this stage as they were book reviews, conference proceedings, reports, editorials, or other non-refereed publications. The third step filtered articles by screening full texts; 54 articles were excluded because they made only passing reference to the FC or were not related to higher education. This meticulous selection yielded a corpus of 233 articles suitable for in-depth analysis, each of which was scrutinized by the authors to confirm its suitability for inclusion. During the selection process, the 233 articles were also systematically categorized into two groups: 131 non-empirical and 102 empirical studies. The non-empirical studies were further divided into two subcategories. The first type was literature reviews; the second type was those drawing on personal observations, reflections on current events, or the authority or experience of the author (Dan, 2021 ). The empirical studies used a variety of systematic methods of collecting materials and analyzing data, including quantitative methods (e.g., survey, correlational research, experimental research) and/or qualitative methods (e.g., interview, case study, record keeping, observation, ethnographic research) (Dan, 2021 ).

Data charting and collation

The fourth stage of Arksey and O’Malley’s scoping review framework is the charting of the selected articles. Summaries of each study were developed. for all studies, these summaries included the author, year of publication, citations per year, foreign language taught, and a brief description of the outcomes. For empirical sources, details related to the research design, study population, and sample size were also provided. Tables 3 and 4 list the top ten most-cited non-empirical and empirical sources. In Table 4 , which references experimental and control groups in results summaries, the experimental group (EG) was the group that took courses in the FC model, while the control group (CG) took courses in a traditional classroom.

Results and analysis

In accordance with the fifth stage of Arksey and O’Malley’s framework for a scoping review, the findings from the 233 included studies are summarized and discussed in the following three sections. Section 4.1 summarizes basic information regarding the included studies; section 4.2 presents a holistic analysis of the research foci and trends over time using keyword clustering analysis and keyword burst analysis; and section 4.3 offers an in-depth content analysis focusing on the categorization of the included studies and discussion of the major findings.

Basic information on the included studies

Distribution by year of publication.

As Fig. 3 shows, the first studies on FCs in the field of FLT in China emerged in 2013. The number of such studies began to steadily increase and reached a peak in 2016 and 2017. Although there was some decrease after that, the FC model has continued to attract research attention, in line with global trends. According to Akçayir and Akçayir’s (2018) review of the literature on FCs published in Social Sciences Citation Index (SSCI) journals as of 31 December 2016, the first article about the FC was published in 2000, but the second was not published until more than a decade later, in 2012; 2013 was also the year that FC studies became popular among scholars. A possible explanation for this increase in interest is the growing availability of internet technologies and the popularity of online learning platforms, such as MOOCs and SPOCs (Small Private Online Courses), along with the view of the FC as a promising model that can open doors to new approaches in higher education in the new century.

Number of articles published by year.

Distribution by foreign language

Figure 4 shows the distribution of foreign languages discussed in the FC literature. The FC model was mainly implemented in EFL teaching (93%), which reflects the dominance of English in FLT in Chinese higher education. Only five articles discussed the use of FC models in Japanese teaching, while one article was related to French teaching. Ten non-empirical studies (4%) reported the feasibility of FC models in FLT without mentioning a specific foreign language.

Distribution by foreign language type.

Research methods of the included studies

Figure 5 shows a breakdown of the methodologies adopted by the studies included in our review. Among the 131 non-empirical studies, three were literature reviews, while the remaining 128 (55%) were descriptive studies based on the introduction of the FC model, including descriptions of its strengths and associated challenges and discussions of its design and implementation in FLT.

Methodological paradigms.

Of the 102 empirical studies, 60 (26%) used quantitative methods for data collection, eight (3%) used qualitative methods, and 34 (15%) used mixed methods. It is interesting to note that although quantitative methods are more common in FC studies, seven of the top ten most-cited empirical studies (as listed above in Table 4 ) used mixed methods. A potential reason may be that research findings collected with triangulation from various data sources or methods are seen as more reliable and valid and, hence, more accepted by scholars.

A breakdown of the data collection approaches used in the 102 reviewed empirical studies is displayed in Table 5 . It is important to note that most studies used more than one instrument, and therefore, it is possible for percentages to add up to more than 100%. The survey, as a convenient, cost-effective, and reliable research method, was the tool most frequently used to gain a comprehensive picture of the attitudes and characteristics of a large group of learners. Surveys were used in 79 of the 102 studies—73 times with learners and six times with teachers—to explore students’ learning experiences, attitudes, and emotions, as well as teachers’ opinions. Some studies used paper-based surveys, while others used online ones. Interviews with learners were used in 33 studies to provide in-depth information; one study used interviews with teachers. Surveys and interviews were combined in 24 studies to obtain both quantitative and qualitative data. Other research approaches included comparing the test scores between experimental and control groups (used in 25 studies) or using the results of course assessments (17 studies) to investigate the effects of the FC on academic performance. Learners’ self-reports (9 studies) were also used to capture the effects of the FC on learners’ experience and cognitive changes that could not be obtained in other ways, while one study used a case study for a similar purpose. Teachers’ class observations and reflections were used in eight studies to evaluate students’ engagement, interaction, activities, and learning performance.

Holistic analysis of the research foci and the changing trends of the included studies

A holistic analysis of the research foci in studies of FCs in China was conducted using CiteSpace5.8.R3, a software developed by Chaomei Chen ( http://cluster.cis.drexel.edu/~cchen/citespace/ , accessed on 20 February 2022), to conduct a visual analysis of the literature. This software can help conduct co-citation analysis, keyword co-occurrence analysis, keyword clustering analysis, keyword burst analysis, and social network analysis (Chen, 2016 ). In this study, keyword clustering analysis and keyword burst analysis were chosen to capture important themes and reveal changing trends in FC research.

Keyword clustering analysis primarily serves to identify core topics in a corpus. Figure 6 presents a graph of the top ten keyword clusters identified in the included studies. In this graph, the lower the ID number of a given cluster, the more keywords are in that cluster. As shown in the top left corner of Fig. 6 , the value of modularity q is 0.8122, which is greater than the critical value of 0.3, indicating that the clustering effect is good; the mean silhouette value is 0.9412, which is >0.5, indicating that the clustering results are significant and can accurately represent hot spots and topics in FC research (Hu and Song, 2021 ). The top ten keyword clusters include #0翻转课堂 (flipped classroom), #1大学英语 (college English), #2 MOOC, #3教学模式 (teaching model), #4元认知 (metacognition), #5微课 (micro lecture), #6微课设计 (micro lecture design), #7英语教学 (English teaching), #8 SPOC, and #9 POA (production-oriented approach).

The graph of the top ten keyword clusters.

Keyword burst analysis is used to showcase the changes in keyword frequencies over a given period of time. By analyzing the rise and decline of keywords, and in particular, the years in which some keywords suddenly become significantly more prevalent (“burst”), we can identify emerging trends in the evolution of FC research. Figure 7 displays the 11 keywords with the strongest citation bursts. We can roughly divide the evolution of FC research documented in Fig. 7 into two periods. The first period (2014 to 2017) focused on the introduction of the new model and the analysis of its feasibility in FLT. The keywords that underwent bursts in this period included “MOOC”, “自主学习” (independent learning), “模式” (model), “学习模式” (learning model), “教师话语” (teacher discourse), “茶文化” (tea culture), and “可行性” (feasibility). The reason for the appearance of the keyword “tea culture” lies in the fact that three articles discussing the use of FCs in teaching tea culture in an EFL environment were published in the same journal, entitled Tea in Fujian , during this period. The second period (2018–2021) focused on the investigation of the effect of FCs and the design of micro lectures. Keywords undergoing bursts during this period included “互联网+” (internet plus), “课堂环境” (classroom environment), “教学效果” (teaching effect), and “微课设计” (micro lecture design). The latter two topics (“teaching effect” and “micro lecture design”) may continue to be prevalent in the coming years.

Top 11 keywords with the strongest citation bursts.

In-depth content analysis of the included studies

Along with the findings from the keyword clustering analysis and keyword burst analysis, an open coding system was created to categorize the research topics and contents of the 233 articles for in-depth analysis. Non-empirical and empirical studies were classified further into detailed sub-categories based on research foci and findings. It is important to note that some studies reported more than one research focus. For such studies, more than one sub-category or more than one code was applied; therefore, it is possible for percentages to add up to more than 100%. The findings for each category are discussed in detail in the following sections.

Non-empirical studies

The 131 non-empirical studies can be roughly divided into two categories, as shown in Table 6 . The first category, literature reviews, has no sub-categories. The second, descriptive studies, includes discussions of how to use FCs in FLT; descriptions of the process of implementing the FC in FLT; and comparisons between FCs and traditional classes or comparisons of FCs in Chinese and American educational contexts.

The sub-categories of “introduction and discussion” and “introduction and description” in Table 6 comprise 91.6% of the non-empirical studies included in our review. The difference between them lies in that the former is based on the introduction of the FC literature, while the latter is based both on the introduction of the FC literature and exploration of researchers’ teaching experience; the latter might have become qualitative studies if researchers had gone further in providing systematic methods of collecting information or an analysis of the impact of FCs.

Empirical studies

The 102 empirical studies were divided into four categories based on the domain of their reported findings: the effect of FCs on learners; learners’ satisfaction with FCs; factors influencing FCs; or other research foci. Each group was further classified into more detailed sub-categories.

Effect of FCs on learners

Studies on the effect of FCs on learners were divided into two types, as presented in Table 7 : those concerned with the direct effect of FCs on learning performance and those exploring the indirect effect on learners’ perceptions. Eight codes were applied to categorize the direct effect of FCs on learning performance, which was usually evaluated through test scores; 14 codes were used to categorize the indirect effect of FCs on learners’ perceptions, which were usually investigated through surveys or questionnaires. We do not provide percentages for each code in Tables 7 – 9 because, given that the total number of empirical studies is 102, the percentages are almost identical to the frequencies.

The results shown in Table 7 reveal that 84 studies of direct educational outcomes reported that FCs had a positive effect on basic language skills, content knowledge, and foreign language proficiency. Of these, 64 were concerned with the positive effect of FCs on foreign language proficiency, speaking skills, or listening skills. This result might be explained by the features of FCs. The main difference between FCs and traditional classrooms is that the teaching of content in FCs has been removed from the classes themselves and is often delivered to the students through video recordings, which can be viewed repeatedly outside of the class. In-class time can thus be used for discussion, presentations, or the extension of the knowledge provided in the videos. It is evident that students have more opportunities to practice listening and speaking in FCs, and foreign language proficiency is naturally expected. Only three studies reported that FCs had no effect or a negative effect on the development of foreign language proficiency, speaking, listening, and writing skills. Yan and Zhou ( 2021 ) found that after the FC model had been in place for one semester, college students’ reading abilities improved significantly, while there was no significant improvement in their listening and writing abilities. Yin ( 2016 ) reported that after FC had been implemented for one semester, there was no significant difference in college students’ speaking scores.

A total of 96 studies reported positive effects on indirect educational outcomes, including: boosting learners’ motivation, interest, or confidence; enhancing engagement, interaction, cooperation, creativity, independent learning ability, or critical thinking ability; fostering information literacy, learning strategies, learning efficiency, or self-efficacy; or relieving stress or anxiety. The most frequently documented indirect effect of FCs is improvement in students’ independent learning ability. Only one study found that the FC did not significantly increase student interest in the course (Wang, 2015 ). Similarly, only one study found that students’ anxiety in the FC was significantly higher than that in a traditional class (Gao and Li, 2016 ).

Learners’ satisfaction with FCs

Table 8 presents the results regarding learners’ satisfaction with FCs. Nine codes were used to categorize the different aspects of learners’ satisfaction investigated in the 102 empirical studies. Some researchers represented learner satisfaction using the percentage of students choosing each answer on a five-point Likert scale from 1 (not at all satisfied) to 5 (very satisfied), while others used average scores based on Likert scale values. For the purposes of our synthesis of findings, if the percentage is above 60% or the average score is above 3, the finding is categorized as satisfied; otherwise, it is categorized as not satisfied.

The results in Table 8 show that among the nine aspects investigated, teaching approach and learning outcomes were most frequently asked about in the research, and learners were generally satisfied with both. Only one study (Li and Cao, 2015 ) reported significant dissatisfaction; in this case, 76.19% of students were not satisfied with the videos used in college English teaching due to their poor quality.

Factors influencing the effect of FCs

Eleven factors were found to influence the effect of FCs; these are categorized in Table 9 .

The results shown in Table 9 indicate that learners’ foreign language proficiency and self-regulation or self-discipline abilities are two important factors influencing the effect of FCs. Learners with high foreign language proficiency benefited more from FCs than those with low foreign language proficiency (Lv and Wang, 2016 ; Li and Cao, 2015 ; Wang and Zhang, 2014 ; Qu and Miu, 2016 ; Wang and Zhang, 2013 ; Cheng, 2016 ; Jia et al., 2016 ; Liu, 2016 ), and learners with good self-regulation and self-discipline abilities benefited more than those with limited abilities (Wang and Zhang, 2014 ; Lu, 2014 ; Lv and Wang, 2016 ; Dai and Chen 2016 ; Jia et al. 2016 ; Ling, 2018 ). It is interesting to note that two studies explored the relationship between gender and FCs (Wang and Zhang, 2014 ; Zhang and He, 2020 ), and both reported that girls benefited more from FCs because they were generally more self-disciplined than boys.

Studies with other research foci

There were six studies with other research foci, three of which investigated teachers’ attitudes toward FCs (Liao and Zou, 2019 ; Zhang and Xu, 2018 ; Zhang et al., 2015 ). The results of the surveys in these three studies showed that teachers generally held positive attitudes towards FCs and felt that the learning outcomes were better than those of traditional classes. However, some problems were also revealed in these studies. First, 56% of teachers expressed the desire to receive training before using FCs due to a lack of theoretical and practical expertise regarding this new model. Second, 87% of teachers thought that the FC increased their workload, as they were spending a significant amount of time learning to use new technology and preparing online videos or materials, yet no policy was implemented in the schools to encourage them to undertake this work. Third, 72% of teachers felt that the FC increased the academic burden students faced in their spare time (Zhang and Xu, 2018 ; Zhang et al., 2015 ). The final three studies include Cheng’s ( 2016 ) investigation of the mediative functions of college EFL teachers in the FC, Wang and Ma’s ( 2017 ) construction of a model for assessing the teaching quality of classes using the FC model, and Luo’s ( 2018 ) evaluation of the learning environment of an FC-model college English MOOC.

Discussion and conclusions

This investigation employed literature visualization to systematically analyze 233 research papers sourced from CSSCI and CJC in the CNKI database, thereby conducting a scoping review delineating the landscape of FC research within the domain of FLT in the context of higher education in China.

Our findings in relation to RQ1 highlight a substantial surge in the number of articles relating to FCs in FLT between 2013 and 2017, followed by a discernible, albeit moderate, decrease. Despite this trend, FC studies continue to be of significant interest to foreign language educators and researchers. This may be attributed to Chinese government policies encouraging higher education reform, increased internet access among educators and learners, and the burgeoning popularity of online courses such as MOOCs and SPOCs. However, the majority of the reviewed FC studies were conducted in college English classes, with only 6 studies on classes teaching foreign languages other than English. It seems that foreign language education in China (and in much of the world) has become synonymous with the teaching and learning of English, with other languages occupying a marginal position, struggling to find space in educational programs. In a multilingual world in which each language offers different possibilities for understanding others, their cultures, their epistemologies, and their experiences, this monolingual approach to FLT is dangerous (Liddicoat, 2022 ). The promotion of linguistic diversity in foreign language education policies and research is thus imperative. Another gap that needs to be addressed is the paucity of studies on the implementation of FCs in adult education. The FC model is expected to be potentially effective for teaching adult learners because it is similar in some respects to online distance learning.

In answer to RQ2, we found that the commonly used research methods and instruments in studies of the FC model include surveys, interviews, comparisons of academic measures between EGs and CGs, and course assessments. The case study is the least used method, likely due to limitations such as time demand, researcher bias, and the fact that it provides little basis for the generalization of results to the wider population. However, more case studies are needed in future research on FCs because they can provide detailed and insightful qualitative information that cannot be gathered in other ways.

Our findings regarding RQ3 show that research foci within the FC domain have evolved over time from initial exploration and feasibility discussions to a subsequent focus on the design of FCs incorporating micro-lectures based on MOOC or SPOC structures, and then to the present focus on the examination of FCs’ impacts on learners. The results of the keyword burst analysis indicate that these thematic areas are likely to persist as prominent subjects of research interest for the foreseeable future.

In response to RQ4, our in-depth content analysis found that FCs, on the whole, yield positive outcomes, although isolated studies identify limited negative impacts. FCs are most frequently associated with enhancements in student learning performance, fostering independent learning, promoting engagement and cooperation, and mitigating stress or anxiety. The results of this study suggest that well-designed FCs present a significant opportunity for foreign language educators to revolutionize instructional approaches. Furthermore, well-structured FCs can facilitate the development of learners’ potential while concurrently enabling the seamless integration of digital technology into FLT.

Most learners are satisfied with FCs, particularly with the innovative pedagogical approach of reversing traditional classes. FCs are perceived as beneficial for improving learning outcomes, creating an environment conducive to peer interaction, and gaining immediate teacher feedback and support. In addition, students’ interest in classes is enhanced by the rich and diverse online learning materials uploaded by teachers, which can be accessed conveniently at any time in any place. Furthermore, the dynamic and formative online assessment approach is also welcomed by students because it provides immediate feedback and the ability to discuss any problems they have with teachers or peers online or offline.

However, it is worth noting that most of the reviewed studies on FCs focused on one course, usually over only one semester. Students’ increase in motivation or improvements in learning outcomes might, therefore, be a result of the Hawthorne effect. Compared with the traditional didactic lecture format, the novelty of FCs, when used for the first time, might generate excitement among students, thus increasing their attention and enhancing learning outcomes, but such benefits will diminish over time. Therefore, there is a need to examine whether this model is suitable for large-scale implementation and whether its effects might be sustained over longer periods of implementation.

Learners’ foreign language proficiency and self-regulation or self-discipline abilities are the two key factors influencing the effect of FCs. These two factors are closely related; self-regulation or self-discipline is a prerequisite for successful foreign language learning in FC contexts and plays a crucial role in students’ success in the pre-class sessions for which they are personally responsible. In addition, factors such as learners’ attitudes, expectations of and adaptability to the FC model, the learning tasks and learning environment, the teaching organization and assessment methods, and the learner’s gender also have some impact on the effect of FCs. However, due to the limited number of studies, there is not sufficient evidence to warrant the generalization of any of these effects.

This scoping review highlights some potential challenges that need to be addressed for the effective implementation of FCs.

First, despite the benefits of the FC model, FCs are not equally advantageous to all students due to the self-regulated nature of the model. Many learners have reported difficulties in completing their individual online tasks outside the classroom (Yoon et al., 2021 ). The non-traditional configuration of FCs poses a formidable challenge, particularly for students less inclined to engage in pre-class online activities characterized by a lack of interactivity and for those who are less self-disciplined. Consequentially, students may attend class without having assimilated the pre-assigned material, thereby diminishing the efficacy of this instructional approach. To address this issue, additional support or prompts for students should be provided to remind them of the need to self-regulate their learning. For example, Park and Jo ( 2015 ) employed a learning analytics dashboard displaying visual representations of students’ learning patterns derived from login traces, such as login frequency and interval regularity, within the course’s learning management system. These visual indicators allowed students to monitor their learning engagement and performance in comparison to those of their peers.

Second, a persistent problem with FCs is the inability of students to interact with their peers or receive prompt feedback from instructors after completing independent online learning activities. While some researchers identified a need for teachers to provide immediate online feedback or opportunities for peer discussion, our review of the literature shows that scant attention has been given to this issue. Researchers note that under-stimulation, low perceived control over tasks, and delayed or insufficient feedback in online learning contribute significantly to learner boredom or absenteeism (Yazdanmehr et al., 2021 ; Tao and Gao, 2022 ). Online pedagogical innovations are needed to solve these new problems. For instance, the establishment of online groups employing chat software like QQ or WeChat could facilitate instantaneous feedback or peer interaction through text-based communication, thereby enhancing learners’ satisfaction with FC courses.

Third, despite recognizing the value of FCs in enhancing the learning experience for students, teachers often lack the requisite training to implement FCs effectively. Insights derived from interviews with teachers, as noted in several of the reviewed studies, reveal a pronounced desire for increased opportunities to learn about the underlying theories of FCs and acquire the skills necessary for the translation of FC concepts into pedagogical practice. Specifically, teachers express a need for guidance in creating engaging instructional videos, determining optimal video length to sustain learner interest, and ascertaining the ideal duration for online quizzes to foster optimal learner performance. Further research is required on strategies and technologies that can help teachers produce high-quality videos despite limited time and technical skills. Support from professional communities, institutions, and technology specialists is thus essential for the provision of effective hybrid offline and online instruction.

Fourth, additional research is required to determine whether workloads for students and teachers are increased by the use of FCs. If this is the case, as found in some of the reviewed studies, then the compelling benefits of FCs would be offset by the extra time needed, making it difficult to draw the conclusion that FCs are more efficient than traditional classes. The majority of language teachers, due to limited skills in technology, online environment management, and online interaction, feel too physically and emotionally overworked to expend more time and energy on enhancing teaching effectiveness. With few teachers having excess spare time, the thought of designing and creating new content might discourage even the most enthusiastic teachers.

Finally, robust empirical evidence is needed to evaluate whether FCs can facilitate students’ higher-order thinking through the use of creative technologies and assessment approaches. Constructs such as creativity and critical thinking are not always easily reduced to measurable items on survey instruments or scores on examinations (Haladyna et al., 2002 ).

In conclusion, the insights garnered from this study have the potential to enrich the global discourse on the benefits and limitations of FCs in diverse cultural and linguistic contexts. Our review included literature accessible through CSSCI and CJC in the CNKI database, and while this provides a thorough selection of the Chinese literature on the subject, our search approach may have excluded valuable FC-related papers published in other languages and countries. Consequently, different search criteria might yield different selection and data results. Future researchers are encouraged to undertake more comprehensive literature reviews encompassing broader databases to fill the gaps in our work and to augment the depth and breadth of knowledge in this domain.

Data availability

The raw data for this paper were collected from articles in Chinese Social Sciences Citation Index (CSSCI) journals and A Guide to the Core Journals of China of Peking University (PKU journals) in the database of China National Knowledge Infrastructure (CNKI) ( https://www.cnki.net/ ). The raw data used to support the findings of this study are available from the corresponding author upon request.

Akcayir G, Akcayir M (2018) The flipped classroom: a review of its advantages and challenges. Comput Educ 126:334–345. https://doi.org/10.1016/j.compedu.2018.07.021

Article   Google Scholar  

Arksey H, O’Malley L (2005) Scoping studies: towards a methodological framework. Int J Soc Res Methodol 8:19–32. https://doi.org/10.1080/1364557032000119616

Baker JW (2000) The ‘Classroom Flip’: using web course management tools to become the guide by the side. In: Chambers JA ed Selected papers from the 11th international conference on college teaching and learning. Florida Community College at Jacksonville, Jacksonville, FL

Google Scholar  

Bergmann J, Sams A (2012) Flip your classroom: reach every student in every class every day. International Society for Technology in Education, Eugene, OR

Brinks-Lockwood R (2014) Flip it! Strategies for the ESL classroom. University of Michigan Press, Ann Arbor, MI

Book   Google Scholar  

Cao P (2020) Construction of blended learning and evaluation of its effect based on a flipped class approach in an ESP course. Foreign Lang World 6:87–94

Chan L, Woore R, Molway L, Mutton T (2022) Learning and teaching Chinese as a foreign language: a scoping review. Rev Educ 10:1–35. https://doi.org/10.1002/rev3.3370

Chen C (2016) CiteSpace: a practical guide for mapping scientific literature. Nova Science Publishers, Hauppauge, NY, USA

Cheng Y (2014) Facing the challenge and flipping ourselves—opportunities and challenges faced by foreign language teachers under the new paradigm of education. Technol Enhanc Foreign Lang 157:44–47

Cheng X (2016) An empirical study of implementations of mediative functions of college EFL teachers in flipped classrooms. Technol Enhanc Foreign Lang 168:48–53

Cui Y, Wang Y (2014) Flipped class model and its application to college English teaching. China Educ Technol 334:116–121

Dan V (2021) Empirical and nonempirical methods. https://www.ls1.ifkw.uni-muenchen.de/personen/wiss_ma/dan_viorela/empirical_and_non_empirical.pdf . Accessed 23 Aug 2021

Daudt HM, van Mossel C, Scott SJ (2013) Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. BMC Med Res Methodol 13:48, http://www.biomedcentral.com/1471-2288/13/48

Article   PubMed   PubMed Central   Google Scholar  

Dai C, Chen J (2016) An analysis of contributing factors of college English flipped classroom with MOOC philosophy. Technol Enhanc Foreign Lang 172:35–41

Davis K, Drey N, Gould D (2009) What are scoping studies? A review of the nursing literature. Int J Nurs Stud 46:1386–1400. https://doi.org/10.1016/j.ijnurstu.2009.02.010

Article   PubMed   Google Scholar  

Deng D (2016) A review of the research on the application of the flipped classroom model in college English teaching. Foreign Lang World 175:89–96

Dou J, Wen S (2015) The teaching reform exploration of college English flipped classroom based on APP. Heilongjiang Res High Educ 5:162–167

Gao Z, Li J (2016) English classroom anxiety of Chinese learners: flipped vs. traditional. Technol Enhanc Foreign Lang 167:37–42

Garg AX, Hackam D, Tonelli M (2008) Systematic review and meta-analysis: When one study is just not enough. Clin J Am Soc Nephrol 3:253–260. https://doi.org/10.2215/CJN.01430307

Haladyna T, Downing S, Rodriguez M (2002) A review of multiple-choice item writing guidelines for classroom assessment. Appl Meas Educ 15:309–334. https://doi.org/10.1207/S15324818AME15035

Hillman S, Selvi AF, Yazan B (2020) A scoping review of world Englishes in the Middle East and North Africa. World Englishes 40:1–17. https://doi.org/10.1111/weng.12505

Hu J, Wu Z (2014) An empirical study on the MOOC-based college English flipped classroom instructional model. Technol Enhanc Foreign Lang 160:40–45

Hu Z, Song J (2021) Research hotspots and trends of online and offline hybrid teaching in China—bibliometric analysis of journal literature based on CNKI (2015–2020). In: 2021 2nd international conference on education, knowledge and information management. IEEE. https://doi.org/10.1109/ICEKIM52309.2021.00032

Hung HT (2014) Flipping the classroom for English language learners to foster active learning. Comput Assist Lang Learn 28:81–96. https://doi.org/10.1080/09588221.2014.967701

Jarvis W, Halvorson W, Sadeque S, Johnston S (2014) A large class engagement(LCE) model based on service-dominant logic (SDL) and flipped classrooms. Educ Res Perspect 41:1–24. http://www.erpjournal.net/

Jia L, Zhang G, Shi C (2016) Flipping medical English reading class based on Wechat public platform and Wechat community. Technol Enhanc Foreign Lang 168:65–69

Jiang Q, Tao Y (2018) The application of “flipped classroom” in teaching MTI translation theory and its effect analysis. Foreign Lang Educ 5:70–74

ADS   Google Scholar  

Jiang Y, Hu J (2018) A study on the large-scale instruction mechanism of SPOC-based college English flipped classroom. Technol Enhanc Foreign Lang 182:9–15

Khan S (2012) The one world schoolhouse: education reimagined. Hodder and Stoughton, London

Lee G, Wallace A (2018) Flipped learning in the English as a foreign language classroom: outcomes and perceptions. TESOL Q 52:62–84

Levac D, Colquhoun H, O’Brien K (2010) Scoping studies: advancing the methodology. Implement. Sci. 5(69):1–9

Li X, Cao H (2015) Research on flipped classroom based on micro-lecture—a case study in college English teaching through videos. Mod Educ Technol 9:70–76. https://doi.org/10.3969/j.issn.1009-8097.2015.09.011

Li X, Wang B (2017) An empirical study on college English flipped classroom model based on context awareness. Technol Enhanc Foreign Lang 178:71–77

Li J, Wu Z (2015) Practice and reflections on flipped college English class. Foreign Lang China 6:4–9. https://doi.org/10.13564/j.cnki.issn.1672-9382.2015.06.002

Liao G, Zou X (2019) Optimizing the teaching design to develop learners’ autonomous study. Educ Res Mon 10:105–111

Liddicoat AJ (2022) Language planning for diversity in foreign language education. Curr Issues Lang Plan 23(5):457–465. https://doi.org/10.1080/14664208.2022.2088968

Ling R (2018) Research on Japanese pronunciation teaching with the flipped classroom model. Jpn Learn Res 4:68–76. https://doi.org/10.13508/j.cnki.jsr.2018.04.010

Article   ADS   Google Scholar  

Liu Y (2016) Research on the construction of flipped classroom model for vocabulary instruction and its effectiveness. Technol Enhanc Foreign Lang 167:43–49

Liu Z, Wu Q (2015) Cultivation of college students’ learning autonomy from the perspective of “Flipped Classroom”. Mod Educ Technol 11:67–72. https://doi.org/10.3969/j.issn.1009-8097.2015.11.010

Lu H (2014) Feasibility analysis on the application of micro-class based “Flipped Classroom” mode in college English teaching. Technol Enhanc Foreign Lang 158:33–36

Luo S (2018) Evaluating the learning environment of MOOC-based college English flipped classrooms. Technol Enhanc Foreign Lang 182:16–22

Lv T, Wang N (2016) A study on the effect of the flipped classroom based on SPOC+teaching resource platform. China Educ Technol 352:85–90

Mays N, Pope C, Popay J (2005) Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy 1:6–20. https://doi.org/10.1258/1355819054308576

Marzouki OF, Idrissi MK, Bennani S (2017) Effects of social constructivist mobile learning environments on knowledge acquisition: a meta-analysis. Int J Interact Mob Technol 11(1):18–39. https://doi.org/10.3991/ijim.v11i1.5982

Ministry of Education of People’s Republic China (2021) 10-year development plan for education information (2011–2020). http://www.moe.gov.cn/srcsite/A16/s3342/201203/t20120313_133322.html . Accessed 30 Dec 2021

O’ Flaherty J, Philips C (2015) The use of flipped classroom in higher education: a scoping review. Internet High Educ 25:85–95. https://doi.org/10.1016/j.iheduc.2015.02.002

Park Y, Jo I-H (2015) Development of the learning analytics dashboard to support students’ learning performance. J Univers Comput Sci 21(1):110–133. https://doi.org/10.3217/jucs-021-01-0110

Qiao H (2017) Study of blended English learning model based on community of inquiry. Technol Enhanc Foreign Lang 176:43–48

Qu S (2019) A content analysis of researches on EFL flipped classrooms in China’s universities. Technol Enhanc Foreign Lang 187:62–68

Qu Q, Miu R (2016) A research on learning strategies of flipped classroom teaching model. China Educ Technol 350:114–119

Shen Y, Sheng Y (2015) Construction of the flipped college English classroom based on Community of Inquiry. Foreign Lang. World 4:81–89

Steen-Utheim AT, Foldnes NA (2018) Qualitative investigation of student engagement in a flipped classroom. Teach High Educ 23:30–324. https://doi.org/10.1080/13562517.2017.1379481

Su X, Liu S, Ma W (2019) The use of flipped classrooms in Chinese college English teaching: a review. Foreign Lang Lit 35:142–148. 1674-6414(2019) 01-0142-07

Tao J, Gao X (2022) Teaching and learning languages online: challenges and responses. System 107:1–9. https://doi.org/10.1016/j.system.2022.102819

Tullock B, Ortega L (2017) Fluency and multilingualism in study abroad: Lessons from a scoping review. System 71:7–21. https://doi.org/10.1016/j.system.2017.09.019

Vygotsky LS (1978) Mind in society: the development of higher psychological processes. Harvard University Press, Cambridge

Wan M(2016) An empirical study on the application of flipped classroom in college English teaching High Educ Explor 5:69–72. www.cnki.net

Wang H (2015) An action research on interpreting teaching based on the flipped classroom. Chin Transl J 1:59–62

Wang X (2014) College foreign language teaching based on MOOC. Heilongjiang Res High Educ 8:157–159

Wang Y (2016) Research on English learning anxiety of non-English majors under flipped class model. J Guangxi Norm Univ: Philos Soc Sci Ed 4:134–139. https://doi.org/10.16088/j.issn.1001-6597.2016.04.020

Wang N, Chen J, Zhang D (2016) SPOC-based flipped classroom of college English: Construction of an efficient learning model. Technol Enhanc Foreign Lang 169:52–57

Wang G, Ma S (2017) A study on college English classroom “Flip Degree” assessment index. Technol Enhanc Foreign Lang 177:23–28

Wang H, Zhang L (2014) The application of flipped classroom in English teaching. Teach Manag 7:141–144. www.cnki.net

Wang S, Zhang L (2014) A study of college EFL learners’ acceptance towards flipped classroom. Mod Educ Technol 3:71–78. https://doi.org/10.3969/j.issn.1009-8097.2014.03.010

Article   CAS   Google Scholar  

Wang X, Zhang C (2013) The application research of flipped classroom in university teaching. Mod Educ Technol 8:11–16. https://doi.org/10.3969/j.issn.1009-8097.2013.08.002

Wang L, Zhao M, Yang W (2018) An empirical study of the instructional model of college oral English flipped classroom based on principles of CDIO engineering education. Technol Enhanc Foreign Lang 180:72–77

Wang Z, Wu M (2017) The analysis of the impact factors on students’ learning behaviors under flipped classroom model. Technol Enhanc Foreign Lang 177:29–34

Wu L (2015) A study on the flipped classroom model in teaching listening and speaking for academic English for general purposes. E-Educ Res 11:81–87. https://doi.org/10.13811/j.cnki.eer.2015.11.013

Xie P (2020) Applying MOOC-based blended learning to the courses of English education. Foreign Lang Educ China 2:43–49

Xu H (2017) An explorative study of college students’ critical thinking under flipped classroom. Technol Enhanc Foreign Lang 173:29–33

Xu T, Li X (2014) Exploration to project-based flipped classroom—new concept, method and technology in English teaching: a case study based on new college English. Foreign Lang China 15:81–87. https://doi.org/10.13564/j.cnki.issn.1672-9382.2014.05.011

Yan J, Zhang W, Yu Y (2016) School-based flipped classroom teaching reform on video-aural-oral course of college English. Mod Educ Technol 2:94–99. https://doi.org/10.3969/j.issn.1009-8097.2016.02.014

Yan Z, Zhou P(2021) Effect of flipped teaching on college students’ EFL acquisition. Foreign Language Learn Theory Pract 2:86–96

Yazdanmehr E, Shirvan ME, Saghafi K (2021) A process tracing study of the dynamic patterns of boredom in an online L3 course of German during COVID-19 pandemic. Foreign Lang Ann 54(3):714–739. https://doi.org/10.1111/flan.12548

Yin H (2016) An empirical study and critical reflection on the flipped classroom model. J Res Educ Ethn Minor 1:25–30

Yoon M, Hill J, Kim D (2021) Designing supports for promoting self-regulated learning in the flipped classroom. J Comput High Educ 33:398–418. https://doi.org/10.1007/s12528-021-09269-z

Yu P (2014) The application of flipped classroom in English teaching. Teach Manag 9:62–64. www.cnki.net

Yu Z, Chen W (2016) The Influence of clicker-aided college English flipped classroom on meta-cognition, cognitive loads and learning achievements. Technol Enhanc Foreign Lang 170:32–37

Zhai X, Lin L (2014) Factors analysis of Chinese learners’ satisfaction in western flipped classroom model (FCM) teaching—an empirical study based on college English. China Educ Technol 327:104–109

Zhang D (2021) Construction and application of a blended golden course framework of college English. Technol Enhanc Foreign Lang 197:71–77

Zhang M, Deng L(2021) Construction and application of “emergent flipped learning” teaching model in the integration of Chinese culture into college English teaching Foreign Language Learn Theory Pract 1:61–70

Zhang M, He X (2020) A study on the adaptability of blended college English learning. Technol Enhanc Foreign Lang. 4:89–94

Zhang M, Sun X (2015) MOOC-based flipped classroom teaching model for follow-up college English courses. Mod Educ Technol 8:81–87. https://doi.org/10.3969/j.issn.1009-8097.2015.08.01

Zhang J, Li K, Du X (2015) College English classroom: based on the survey of the current situation. Mod Educ Technol 7:68–74. https://doi.org/10.3969/j.issn.1009-8097.2015.07.011

Zhang Y, Xu Z (2018) Research on the acceptance of college English teachers’ flipped classroom. Theory Pract Educ 36:53–55

ADS   CAS   Google Scholar  

Zhou P (2015) Flipped classroom based on modern educational technology and its theoretical foundations. Technol Enhanc Foreign Lang 162:72–76

Zhu L, Xu Y, Han J (2021) An approach for the integration of foreign language teaching and information technology. Foreign Lang World 2:46–62

Download references

Acknowledgements

This research was funded by The 14th Five-year Plan for Education Science of Jiangsu Province (Grant number: D/2021/01/79), Changzhou University (Grant number: GJY2021013), and Department of Education of Zhejiang Province, China (Project of Ideological and Political Construction of Courses 2021-337).

Author information

Authors and affiliations.

Department of English, Changzhou University, 213100, Changzhou, Jiangsu Province, China

School of English Education, Guangdong University of Foreign Studies, 510006, Guangzhou, Guangdong Province, China

College of Science & Technology, Ningbo University, 315200, Ningbo, Zhejiang Province, China

Quanjiang Guo

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization, WK and Q-JG; methodology, WK; software, DL; validation, WK, DL, and Q-JG; formal analysis, WK and Q-JG; investigation, WK and Q-JG; resources, DL; data curation, DL; writing—original draft preparation, WK and Q-JG; writing—review and editing, WK and Q-JG; visualization, WK and Q-JG; supervision, WK; project administration, WK; funding acquisition, WK and Q-JG. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Quanjiang Guo .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

Ethical approval was not required as the study did not involve human participants.

Informed consent

Informed consent was not required as the study did not involve human participants.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Kong, W., Li, D. & Guo, Q. Research on flipped classrooms in foreign language teaching in Chinese higher education. Humanit Soc Sci Commun 11 , 525 (2024). https://doi.org/10.1057/s41599-024-03019-z

Download citation

Received : 17 October 2023

Accepted : 02 April 2024

Published : 23 April 2024

DOI : https://doi.org/10.1057/s41599-024-03019-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

• Publications
• Our Methods
• Short Reads
• Tools & Resources

Read Our Research On:

About 1 in 5 U.S. teens who’ve heard of ChatGPT have used it for schoolwork

Roughly one-in-five teenagers who have heard of ChatGPT say they have used it to help them do their schoolwork, according to a new Pew Research Center survey of U.S. teens ages 13 to 17. With a majority of teens having heard of ChatGPT, that amounts to 13% of all U.S. teens who have used the generative artificial intelligence (AI) chatbot in their schoolwork.

Teens in higher grade levels are particularly likely to have used the chatbot to help them with schoolwork. About one-quarter of 11th and 12th graders who have heard of ChatGPT say they have done this. This share drops to 17% among 9th and 10th graders and 12% among 7th and 8th graders.

There is no significant difference between teen boys and girls who have used ChatGPT in this way.

The introduction of ChatGPT last year has led to much discussion about its role in schools , especially whether schools should integrate the new technology into the classroom or ban it .

Pew Research Center conducted this analysis to understand American teens’ use and understanding of ChatGPT in the school setting.

The Center conducted an online survey of 1,453 U.S. teens from Sept. 26 to Oct. 23, 2023, via Ipsos. Ipsos recruited the teens via their parents, who were part of its KnowledgePanel . The KnowledgePanel is a probability-based web panel recruited primarily through national, random sampling of residential addresses. The survey was weighted to be representative of U.S. teens ages 13 to 17 who live with their parents by age, gender, race and ethnicity, household income, and other categories.

This research was reviewed and approved by an external institutional review board (IRB), Advarra, an independent committee of experts specializing in helping to protect the rights of research participants.

Here are the  questions used for this analysis , along with responses, and its  methodology .

Teens’ awareness of ChatGPT

Overall, two-thirds of U.S. teens say they have heard of ChatGPT, including 23% who have heard a lot about it. But awareness varies by race and ethnicity, as well as by household income:

• 72% of White teens say they’ve heard at least a little about ChatGPT, compared with 63% of Hispanic teens and 56% of Black teens.
• 75% of teens living in households that make $75,000 or more annually have heard of ChatGPT. Much smaller shares in households with incomes between $30,000 and $74,999 (58%) and less than $30,000 (41%) say the same.

Teens who are more aware of ChatGPT are more likely to use it for schoolwork. Roughly a third of teens who have heard a lot about ChatGPT (36%) have used it for schoolwork, far higher than the 10% among those who have heard a little about it.

When do teens think it’s OK for students to use ChatGPT?

For teens, whether it is – or is not – acceptable for students to use ChatGPT depends on what it is being used for.

There is a fair amount of support for using the chatbot to explore a topic. Roughly seven-in-ten teens who have heard of ChatGPT say it’s acceptable to use when they are researching something new, while 13% say it is not acceptable.

However, there is much less support for using ChatGPT to do the work itself. Just one-in-five teens who have heard of ChatGPT say it’s acceptable to use it to write essays, while 57% say it is not acceptable. And 39% say it’s acceptable to use ChatGPT to solve math problems, while a similar share of teens (36%) say it’s not acceptable.

Some teens are uncertain about whether it’s acceptable to use ChatGPT for these tasks. Between 18% and 24% say they aren’t sure whether these are acceptable use cases for ChatGPT.

Those who have heard a lot about ChatGPT are more likely than those who have only heard a little about it to say it’s acceptable to use the chatbot to research topics, solve math problems and write essays. For instance, 54% of teens who have heard a lot about ChatGPT say it’s acceptable to use it to solve math problems, compared with 32% among those who have heard a little about it.

Note: Here are the  questions used for this analysis , along with responses, and its  methodology .

• Artificial Intelligence
• Technology Adoption
• Teens & Tech

Olivia Sidoti is a research assistant focusing on internet and technology research at Pew Research Center

Jeffrey Gottfried is an associate director focusing on internet and technology research at Pew Research Center

Many Americans think generative AI programs should credit the sources they rely on

Americans’ use of chatgpt is ticking up, but few trust its election information, q&a: how we used large language models to identify guests on popular podcasts, striking findings from 2023, what the data says about americans’ views of artificial intelligence, most popular.

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 |  Media Inquiries

Research Topics

• Age & Generations
• Coronavirus (COVID-19)
• Economy & Work
• Family & Relationships
• Gender & LGBTQ
• Immigration & Migration
• International Affairs
• Internet & Technology
• Methodological Research
• News Habits & Media
• Non-U.S. Governments
• Other Topics
• Politics & Policy
• Race & Ethnicity
• Email Newsletters

ABOUT PEW RESEARCH CENTER  Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of  The Pew Charitable Trusts .

Copyright 2024 Pew Research Center

Terms & Conditions

Privacy Policy

Cookie Settings

Reprints, Permissions & Use Policy