literature review using chatgpt

How To Use ChatGPT To Write A Literature Review: Prompts & References

In the rapidly evolving world of academic research, the integration of AI tools like ChatGPT has transformed the traditional approach to literature reviews. As a researcher, you should also leverage on this tool to make your research work easier.

In this post, explore how ChatGPT can enhance the literature review process. We’ll explore how specific prompts can effectively guide this advanced AI model to generate insightful content, while ensuring accuracy, relevance, and academic integrity in their scholarly work.

How to Use ChatGPT for Writing a Literature Review

Understanding chatgpt’s limitations.

While it can efficiently generate content, streamline the research process, and provide a comprehensive understanding of relevant literature, its capabilities are not without constraints. Here are some for you to consider:

Dependence On Pre-Existing Datasets

Since ChatGPT is a language model trained on available data, it may not include the most recent research papers or cutting-edge findings in a specific field. This gap can lead to a lack of current state-of-research insights, particularly crucial in fields like technology and science where advancements happen rapidly.

May Lack Depth And Context

ChatGPT, while able to produce summaries and synthesize information, might not fully grasp the nuanced arguments or complex theories specific to a research topic. This limitation necessitates that researchers critically evaluate and supplement AI-generated text with thorough analysis and insights from recent systematic reviews and primary sources.

Risk Of Plagiarism

Although ChatGPT can generate human-like text, it’s vital to ensure that the content for your literature review is original and properly cited. Relying solely on ChatGPT to write a literature review defeats the purpose of engaging deeply with the material and developing a personal understanding of the literature.

Not A Total Replacement of A Researcher

While ChatGPT can assist non-native English speakers in crafting clear and concise academic writing, it’s not a replacement for the human ability to contextualize and interpret research findings. Researchers must guide the AI model with specific prompts and leverage it as a tool rather than a substitute for comprehensive analysis.

By keeping these limitations in mind, ChatGPT can be a valuable aid in the literature review process, but it should be used judiciously and in conjunction with traditional research methods.

Defining Research Objective

When starting on writing a literature review, the initial step involves using ChatGPT to define your research question or hypothesis.

The AI model’s ability to respond with a summary of the current state of research in your field can provide a comprehensive understanding, especially for systematic reviews or research papers.

For example, by inputting a prompt related to your research topic, ChatGPT can generate human-like text, summarizing prior research and highlighting relevant literature.

One insider tip for effectively using ChatGPT in the literature review process is to leverage its natural language processing capabilities to identify relevant keywords.

These keywords are crucial for non-native English speakers or those new to a research field, as they streamline the search for pertinent academic writing. Additionally, ChatGPT can guide you in understanding the context of your research topic, offering insights that are often challenging to find.

Using AI language models like ChatGPT for generating content for your literature review is efficient and effective, saving valuable time. However, it’s vital to critically evaluate the generated text to ensure it aligns with your research objectives and to avoid plagiarism.

ChatGPT’s ability to synthesize large amounts of information can aid in developing a clear and concise outline, but remember, it’s a guide, not a replacement for human analysis.

Despite these limitations, ChatGPT provides a unique advantage in conducting literature reviews. It can automate mundane tasks, allowing researchers to focus on analysis and critical thinking.

Identifying Keywords and Search Terms

Using ChatGPT to identify relevant keywords related to your research topic can significantly streamline your workflow.

For instance, when you input a summary of your research question into ChatGPT, the AI model can generate a list of pertinent keywords.

These keywords are not just randomly selected; they are based on the vast amounts of information in ChatGPT’s dataset, making them highly relevant and often inclusive of terms that are current in your research field.

An insider tip for leveraging ChatGPT effectively is to guide the AI with clear and concise prompts.

For example, asking ChatGPT to: “summarize key themes in [specific field] research papers from the last five years” can yield a list of keywords and phrases that are not only relevant but also reflective of the current state of research.

This approach is particularly beneficial for conducting systematic reviews or for non-native English speakers who might be unfamiliar with specific academic jargon.

While ChatGPT can provide a comprehensive understanding of relevant literature and help automate the identification of keywords, it’s important to critically evaluate the generated content.

Researchers should use ChatGPT as a tool to augment their research process, not as a replacement for human insight.

It’s crucial to mind the limitations of the AI model and ensure that the keywords identified align with the research topic and objectives.

Creating an Initial Literature Review Outline

The key to using ChatGPT effectively in crafting an initial outline lies in its ability to generate content based on specific prompts.

For instance, a researcher working on organic photovoltaic devices can input a prompt into ChatGPT, such as “Help me create a structure for a literature review on organic photovoltaic devices.”

The AI model, using its comprehensive understanding of the research topic, can then produce a preliminary structure, including sections like:

• Introduction
• Advances in materials and technology, performance, and efficiency.

This generated outline serves as a valuable starting point. It helps in organizing thoughts and determining the key areas that the literature review should cover. I

mportantly, researchers can refine and expand this initial outline as they delve deeper into their topic, ensuring it aligns with their specific research question and the current state of research.

However, while ChatGPT can streamline the review process and save valuable time in creating an initial outline, researchers should not solely rely on it.

The content generated by ChatGPT must be critically evaluated and supplemented with in-depth research. This involves:

• Reading systematic reviews
• Reading research papers, and
• Summarizing relevant literature to ensure the review is comprehensive and up-to-date.

Get ChatGPT To Help You During Research, Using The Right Prompts

The key to effectively using ChatGPT in this process lies in crafting the right prompts, guiding the AI to generate relevant and useful content. 

When initiating a literature review, the prompt should aim for a broad understanding of the research topic. For instance, asking ChatGPT to:

• “Give a brief overview of research done on [topic]”
• “What are some of the recent findings on the [topic] in research?” or 
• “Summarize the historical development of [topic] in academia”

Helps in capturing the general landscape of the field. These prompts assist in identifying key theories, methodologies, and authors within the research area. As the review progresses, more specific prompts are necessary to delve deeper into individual studies. Queries like:

• “Summarize the main arguments and findings of [specific paper]” or
• “What are the strengths and weaknesses of [specific paper]?”

enable ChatGPT to provide detailed insights into particular research papers, aiding in understanding their contribution to the broader field. Comparative prompts are also crucial in synthesizing information across multiple works. Asking ChatGPT to:

• “Compare and contrast the methodologies of [paper 1] and [paper 2]” or
• “How do the findings of [paper 1] and [paper 2] agree or disagree?”

helps in discerning the nuances and disparities in the literature. In the final stages of the literature review, prompts should focus on summarizing findings and identifying emerging trends or gaps. For example:

• “What trends or patterns have emerged from the literature on [topic]?” or
• “What future research directions are suggested by the literature on [topic]?”

We will share more on these ChatGPT prompts in the later part of this post, read on.

Reviewing ChatGPT’s Responses

When using ChatGPT to write a literature review, it’s crucial to critically evaluate its responses.

Firstly, researchers should cross-reference the information provided by ChatGPT with actual research papers.

This step ensures the accuracy of the data and helps in identifying any discrepancies or outdated information, given that ChatGPT’s dataset may not include the most recent studies.

Another essential aspect is assessing the coherence and depth of the AI-generated text. ChatGPT can summarize and synthesize information efficiently, but it might not capture the nuances of complex theories or research arguments.

Researchers should ensure that the content aligns with their research question and systematically reviews the topic comprehensively. This is where a researcher’s value comes in.

Additionally, verifying the originality of the content is vital to avoid plagiarism. While ChatGPT can generate human-like text, researchers must ensure that the AI-generated content is used as a guide rather than a verbatim source. 

Proper citations and references are essential to maintain the integrity of the literature review. Avoid torpedoing your own research by committing plagiarism.

Ensuring Coherence and Flow

One of the challenges when using such advanced AI language models is ensuring the coherence and flow of the final document. This aspect is crucial as it determines the readability and academic rigor of the literature review.

ChatGPT can generate vast amounts of content on a wide range of topics, responding efficiently to prompts and synthesizing information from its extensive dataset.

However, the content generated by ChatGPT, while informative, might not always align seamlessly with the specific research question or maintain a consistent narrative flow.

To tackle this, researchers need to take an active role in guiding ChatGPT and subsequently refining its output.

A practical approach is to use ChatGPT as a starting point, leveraging its ability to quickly provide summaries, synthesize relevant literature, and identify key references and keywords related to the research topic. For example, prompts like:

• “Summarize the current research on [topic]” or
• “Identify key debates in [topic]”

Can yield valuable initial insights.

Once this foundational information is obtained, the crucial task is to carefully review and edit the AI-generated content.

This involves connecting the dots between different sections, ensuring that each part contributes meaningfully to addressing the research question, and refining the language to maintain academic standards.

It’s also essential to check for and avoid plagiarism, ensuring that all sources are correctly cited.

In addition, considering the vast amounts of information ChatGPT can access, it’s vital to verify the accuracy and relevance of the content.

Researchers should cross-reference AI-generated summaries with actual research papers, especially the most recent ones, as ChatGPT’s dataset may not include the latest studies.

Editing and Proofreading

Now that your literature review is mostly written out, now focus on the editing and proofreading. The content generated by ChatGPT needs to be meticulously reviewed and edited. Here are the steps:

• Verifying the accuracy of the information. Researchers must cross-check the AI-generated content against actual research papers and systematic reviews. This ensures that the latest studies are accurately represented.
• Improve coherence and flow. Researchers should restructure sentences, ensure logical progression of ideas, and maintain a consistent academic tone throughout the document.
• Checking for plagiarism. Despite ChatGPT’s ability to generate human-like text, researchers must ensure that all sources are correctly cited and that the review does not inadvertently replicate existing material.
• Check Grammar and Spelling: Editing should encompass grammar checks, vocabulary refinement, and ensuring that the language used is appropriate for an academic audience.
• Update Citation: Review citation, or reference list to ensure everything is cited correctly, and the citation list is written out to your required standard, be it MLA, Chicago, or APA.

What ChatGPT Prompts To Use When Writing A Literature Review?

There are many ways to use ChatGPT to write literature review, usually by using the right prompts. Here’s how specific types of prompts can be effectively employed, with multiple examples for each category:

• “Provide a comprehensive overview of the latest research on [topic].”
• “Summarize the current understanding and key findings in the field of [topic].”
• “Detail the dominant theoretical frameworks currently used in [topic].”
• “Describe the evolution of theoretical approaches in [topic] over the past decade.”
• “Identify and discuss the major debates or controversies in [topic].”
• “What are the conflicting viewpoints or schools of thought in [topic]?”
• “List the leading researchers in [topic] and summarize their key contributions.”
• “Who are the emerging authors in [topic], and what unique perspectives do they offer?”
• “Explain the most common research methodologies used in studies about [topic].”
• “How have the methodologies in [topic] research evolved recently?”
• “Trace the historical development and major milestones in [topic].”
• “Provide a timeline of the key discoveries and shifts in understanding in [topic].”
• “What significant paradigm shifts have occurred in [topic] in the last twenty years?”
• “How has the focus of research in [topic] changed over time?”
• “Analyze the methodology and conclusions of [specific paper].”
• “Discuss the impact and reception of [specific paper] in the field of [topic].”
• “Compare the results and methodologies of [paper 1] and [paper 2] in [topic].”
• “How do [paper 1] and [paper 2] differ in their approach to [topic]?”
• “Based on current literature, what are the suggested future research directions in [topic]?”
• “Identify gaps in the literature of [topic] that could be explored in future studies.”

By using these types of prompts, researchers can guide ChatGPT to produce content that is not only relevant to their literature review but also rich in detail and scope.

Wrapping Up: Use Other AI Tools Too, Not Just ChatGPT

In conclusion, while ChatGPT serves as a powerful ally in the literature review process, it’s important to recognize it as one of many AI tools available to researchers as well. Diversifying your AI toolkit can enhance the depth and breadth of your review, offering varied perspectives and methodologies.

As AI continues to evolve, embracing a range of these tools can lead to more comprehensive, nuanced, and innovative academic writing, expanding the horizons of research and scholarly exploration beyond what we currently envision.

Dr Andrew Stapleton has a Masters and PhD in Chemistry from the UK and Australia. He has many years of research experience and has worked as a Postdoctoral Fellow and Associate at a number of Universities. Although having secured funding for his own research, he left academia to help others with his YouTube channel all about the inner workings of academia and how to make it work for you.

Thank you for visiting Academia Insider.

We are here to help you navigate Academia as painlessly as possible. We are supported by our readers and by visiting you are helping us earn a small amount through ads and affiliate revenue - Thank you!

2024 © Academia Insider

• Interesting
• Scholarships
• UGC-CARE Journals

Step-by-Step Guide: How to Use ChatGPT to Write a Literature Review With Prompts

Step-by-Step Guide: How to Use ChatGPT for Writing a Literature Review

Writing a literature review can be a challenging task for researchers and students alike. It requires a comprehensive understanding of the existing body of research on a particular topic. However, with the advent of advanced language models like ChatGPT, the process has become more accessible and efficient.

In this step-by-step guide, ilovephd will explore how you can leverage ChatGPT to write a compelling literature review that is both informative.

A Step-by-Step Guide: How to Use ChatGPT for Writing a Literature Review

Step 1: Defining Your Research Objective Before diving into the literature review process, it is crucial to define your research objective.

Clearly articulate the topic, research question, or hypothesis you aim to address through your literature review. This step will help you maintain focus and guide your search for relevant sources.

Step 2: Identifying Keywords and Search Terms To effectively use ChatGPT to assist in your literature review, you need to identify relevant keywords and search terms related to your research topic.

These keywords will help you narrow down your search and gather pertinent information. Consider using tools like Google Keyword Planner or other keyword research tools to discover commonly used terms in your field.

Step 3: Familiarizing Yourself with ChatGPT Before engaging with ChatGPT, it is essential to understand its capabilities and limitations. Familiarize yourself with the prompts and commands that work best with the model.

Keep in mind that ChatGPT is an AI language model trained on a vast amount of data, so it can provide valuable insights and suggestions, but it’s important to critically evaluate and validate the information it generates.

Step 4: Generating an Initial Literature Review Outline Start by creating an outline for your literature review. Outline the main sections, such as the introduction, methodology, results, discussion, and conclusion.

Within each section, jot down the key points or subtopics you want to cover. This will help you organize your thoughts and structure your review effectively.

Step 5: Engaging with ChatGPT for Research Assistance Once you have your outline ready, engage with ChatGPT for research assistance.

Begin by providing a clear and concise prompt that specifies the topic, context, and any specific questions you have. For example, “What are the current trends in [your research topic]?” or “Can you provide an overview of the main theories on [your research question]?”

Step 6: Reviewing and Selecting Generated Content ChatGPT will generate a response based on your prompt. Carefully review the content generated, considering its relevance, accuracy, and coherence.

Extract key points, relevant references, and insightful arguments from the response and incorporate them into your literature review. Be sure to cite and attribute the sources appropriately.

Step 7: Ensuring Coherence and Flow While ChatGPT can provide valuable content, it’s important to ensure the coherence and flow of your literature review.

Use your critical thinking skills to connect the generated content with your research objective and existing knowledge. Rearrange, rephrase, and expand upon the generated text to ensure it aligns with the structure and purpose of your review.

Step 8: Editing and Proofreading Once you have incorporated the generated content into your literature review, thoroughly edit and proofread the document.

Check for grammatical errors, consistency in referencing, and overall clarity. This step is crucial to ensure your literature review is polished and professional.

ChatGPT prompts to Write a Literature Review

Prompts you can use when engaging with ChatGPT for research assistance in writing a literature review:

• “Can you provide an overview of the main theories and concepts related to [your research topic]?”
• “What are the current trends and developments in [your research field]?”
• “Can you suggest some key studies or research papers on [specific aspect of your research topic]?”
• “What are the main methodologies used in conducting research on [your research topic]?”
• “Can you provide a critical analysis of the existing literature on [your research question]?”
• “Are there any gaps or areas of controversy in the literature on [your research topic] that need further exploration?”
• “What are the key findings and conclusions from the most recent studies on [your research topic]?”
• “Can you suggest some reputable journals or publications explore for relevant literature in [your research field]?”
• “What are the different perspectives or schools of thought in the literature on [your research topic]?”
• “Can you provide a summary of the historical background and evolution of research on [your research topic]?”

Remember to provide clear and specific instructions in your prompts to guide ChatGPT in generating relevant and accurate content for your literature review.

Using ChatGPT to write a literature review can greatly facilitate the research process. By following a step-by-step approach, researchers can effectively leverage ChatGPT’s capabilities to gather insights, generate content, and enhance the quality of their literature review. However, it is important to approach the generated content critically, validate it with reliable sources, and ensure coherence within the review.

• Academic Writing
• AI writing tool
• literature review guide
• literature review tips
• research assistance
• research process
• research writing
• write literature review

24 Best Free Plagiarism Checkers in 2024

How does gptzero work ai detector for chatgpt / gemini / copilot / meta ai, how to effectively use meta ai for phd research, most popular, 100 connective words for research paper writing, phd supervisors – unsung heroes of doctoral students, india-canada collaborative industrial r&d grant, call for mobility plus project proposal – india and the czech republic, effective tips on how to read research paper, iitm & birmingham – joint master program, anna’s archive – download research papers for free, fulbright-kalam climate fellowship: fostering us-india collaboration, fulbright specialist program 2024-25, six effective tips to identify research gap, best for you, what is phd, popular posts, how to check scopus indexed journals 2024, how to write a research paper a complete guide, 480 ugc-care list of journals – science – 2024, popular category.

• POSTDOC 317
• Interesting 258
• Journals 234
• Fellowship 128
• Research Methodology 102
• All Scopus Indexed Journals 92

iLovePhD is a research education website to know updated research-related information. It helps researchers to find top journals for publishing research articles and get an easy manual for research tools. The main aim of this website is to help Ph.D. scholars who are working in various domains to get more valuable ideas to carry out their research. Learn the current groundbreaking research activities around the world, love the process of getting a Ph.D.

Contact us: [email protected]

Google News

Copyright © 2024 iLovePhD. All rights reserved

• Artificial intelligence

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 28 July 2023

Special feature

Application ChatGPT in conducting systematic reviews and meta-analyses

• Simpy Amit Mahuli 1 ,
• Arpita Rai 2 ,
• Amit Vasant Mahuli 3 &
• Ansul Kumar 4  

British Dental Journal volume  235 ,  pages 90–92 ( 2023 ) Cite this article

4715 Accesses

9 Citations

Metrics details

You have full access to this article via your institution.

Systematic reviews and meta-analyses, which involve thorough literature searches and synthesis, are essential for evidence-based decision-making. With the growing number of publications, thorough review takes more time. A systematic review often takes more than 15 months to complete, and the long conception-completion interval may render a systematic review outdated by the time they are ready to be submitted and published. 1 , 2

Artificial intelligence (AI)-based quick approaches are being developed in response to the growing demand for screening and data extraction techniques that are more effective. These techniques automate decision-making by utilising machine learning (ML) algorithms and natural language processing (NLP) to extract pertinent information from textual material. These AI-based methods have the potential to accelerate the processes of systematic reviews and evidence synthesis while requiring less human labour. 3 , 4

ChatGPT is a Large Language Model (LLM) developed by OpenAI (OpenAI, L.L.C., San Francisco, CA, USA) which has tremendous potential to revolutionise the healthcare industry. ChatGPT has been effectively used for a variety of tasks in healthcare, including conducting literature reviews, analysing datasets, writing academic papers, documentation, and enhancing clinical workflow. ChatGPT is a significant advancement in the field of NLP. It generates human-like text by mimicking human language processing abilities using deep learning techniques and neural networks. Owing to its training on diverse datasets, it is able to provide pertinent responses to human prompts by processing large amounts of information. The easy accessibility and free availability have contributed to the success ChatGPT version 3.5 platform. 5

We tried to use ChatGPT for conducting Risk of Bias analysis and data extraction from a randomised controlled trial. The response of ChatGPT was surprising, to say the least. Conducting such processes through ChatGPT is simple and straightforward.

Firstly, to conduct a ROB analysis effectively using AI, obtaining the full-text version of the article under evaluation is crucial. This ensures that all relevant information and details necessary for the assessment are available. As an AI tool, it does not have real-time browsing capabilities, so it cannot search the internet or access the latest research articles. Additionally, it being a text-based AI, we cannot upload or import text/pdf files into the chatbox. However, ChatGPT can still try to assist us with general questions or provide guidance based on the information we provide from the study. If we have any specific questions or excerpts from the study that we would like ChatGPT to analyse or discuss, we can paste them into the chat box, and ChatGPT will do its best to assist.

AI provides validation to the information obtained through human effort and may even catch some details that are missed by the human eye.

Secondly, we need to provide clear and specific instructions to the tool for it to provide an appropriate response. Hence when prompted 'Can you use the ROBINS-I tool to evaluate "(Materials and methods from the full-text article)"', it generated a detailed response highlighting the various lacunae in the methodology in context of the Biases included in the ROBINS-I tool ( Fig. 1 and Fig. 2 ). This assessment does not eliminate the need for a human expert; however, it considerably reduces the time and effort required for the process. Also, AI provides validation to the information obtained through human effort and may even catch some details that are missed by the human eye. Additionally, ChatGPT is extremely helpful in the interpretation of studies for those who are not so fluent in the language. While ChatGPT can comprehend and generate text in languages other than English to some extent, its proficiency may vary across different languages. It's worth noting that ChatGPT's performance is generally more robust in English, as the majority of its training data is in English.

Screenshot of query for RoB analyis

Screenshot of ChatGPT's response to RoB analysis

Next, we tried to use ChatGPT for data extraction. By sharing the complete text and specifying the desired information or data to be extracted, AI can be of assistance. Hence, following various prompts from us, ChatGPT extracted data like mean values, standard deviation and sample size for study groups and sub-groups and presented it. Also, it was able to extract information from the tables provided, though it is not possible to paste a table in the chatbox. We can paste the data in the table en masse into the chatbox and it was able to interpret and extract information from it ( Fig. 3 and Fig. 4 ). Needless to say, this will reduce the time required for data extraction as well as ensure accuracy of the values, thereby reducing human errors.

Screenshot of query for data extraction from text

Response from ChatGPT for various queries for data extraction

AI methods are increasingly integrated into healthcare reviews to streamline processes. A recent systematic review by Blaizot et al. identified and examined automated tools and platforms that utilise artificial intelligence (AI) techniques. It assessed the pros and cons associated with the use of these methods in conducting systematic reviews. They identified various tools like Rayyan, Robot Reviewer, EPPI-reviewer, SWIFT-review, and Abstrackr which were employed for different review stages. Most included reviews acknowledged that screening with AI methods generally demonstrates low risks and enhances efficiency. However, possibility of missing relevant articles and issues in data extraction required human intervention. 4

ChatGPT possesses several distinct advantages in comparison to other AI tools. Its ability to retain context and coherence during conversations equips it to comprehend and respond to complex queries. It is extremely versatile which is demonstrated through its proficiency in handling various tasks, including factual inquiries, explanatory responses, creative content generation, and problem-solving assistance. ChatGPT is continuously evolving through regular updates and fine-tuning based on user feedback and new training data, enhancing its performance and accuracy over time. 5

It is crucial to acknowledge that ChatGPT also has limitations. These include the potential for generating incorrect or biased responses and its reliance on pre-existing data. Users should exercise critical thinking and independently verify information obtained from any AI tool. Currently, it is unable to process visual information; however, this issue has been addressed in ChatGPT 4.0. Additionally, the model's output heavily relies on user input; minor changes in the query can lead to significant alterations in the response. 5

This was the first attempt to test the feasibility of using ChatGPT in the tedious task of conducting systematic reviews and meta-analyses. It shows promise in reducing workload and time, but careful implementation and validation are necessary. We as researchers need to interact more with this tool to completely understand its applicability in generating evidence.

Borah R, Brown A W, Capers P L, Kaiser K A. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 2017; doi: 10.1136/bmjopen-2016-012545.

Yaffe J, Montgomery P, Hopewell S, Shepard L D. Empty reviews: a description and consideration of Cochrane systematic reviews with no included studies. PLoS One 2012; doi: doi: 10.1371/journal.pone.0036626.

Chollet F. Deep learning with Python. pp 361. Shelter Island, New York: Manning Publications Co, 2018.

Blaizot A, Veettil S K, Saidoung P, Moreno-Garcia C F et al . Using artificial intelligence methods for systematic review in health sciences: A systematic review. Res Synth Methods 2022; 13: 353-362.

OpenAI. Introducing ChatGPT. Available at: https://openai.com/blog/chatgpt (accessed 19 May 2023).

Download references

Author information

Authors and affiliations.

PhD Scholar, Rajendra Institute of Medical Sciences (RIMS), Bariatu, Ranchi – 834009, India

Simpy Amit Mahuli

Associate Professor, Oral Medicine and Radiology, Rajendra Institute of Medical Sciences (RIMS), Bariatu, Ranchi – 834009, India

Associate Professor, Public Health Dentistry, Rajendra Institute of Medical Sciences (RIMS), Bariatu, Ranchi – 834009, India

Amit Vasant Mahuli

Associate Professor, Cardiothoracic and Vascular Surgery, Dental College, Rajendra Institute of Medical Sciences, Bariatu, India

Ansul Kumar

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Simpy Amit Mahuli , Arpita Rai , Amit Vasant Mahuli or Ansul Kumar .

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Mahuli, S., Rai, A., Mahuli, A. et al. Application ChatGPT in conducting systematic reviews and meta-analyses. Br Dent J 235 , 90–92 (2023). https://doi.org/10.1038/s41415-023-6132-y

Download citation

Published : 28 July 2023

Issue Date : 28 July 2023

DOI : https://doi.org/10.1038/s41415-023-6132-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Ai-assisted boolean search.

• J. M. Cherian
• K. G. Varghese

British Dental Journal (2023)

AI-based article screening

• M. Mehrabanian

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• FIU Libraries
• RefTech LibGuides
• Artificial Intelligence Now: ChatGPT + AI Literacy Toolbox
• Literature Reviews with Prompts

Artificial Intelligence Now: ChatGPT + AI Literacy Toolbox: Literature Reviews with Prompts

• Literacy Tutorials
• AI LibGuides
• AI Literacy
• AI + Art News
• AI Plagiarism + Citation
• Which AI tools should I choose?
• Faculty + Academic Writing
• ChatGPT on InfoLit
• Libraries + AI
• Art + Image Prompts
• Prompt Literacy
• Text Prompts
• Scholarship + Publishing
• Students + AI Use
• FIU Library Resources
• How does ChatGPT work?
• Zoom Recordings: AI Workshops
• ALA 2024: Breaking Boundaries

Resources on Lit Reviews & AI

• How to Use ChatGPT to Accelerate Literature Review I’d like to show you how I use ChatGPT to speed up my literature review process. The topic I am exploring is heterogeneity(diversity) learning. I started with the keyword “Info-GAIL” as I read about it when using GAIL in the past.
• ChatGPT Simple Literature Review Template The evolution of library services in the digital age has seen a significant shift towards automation and artificial intelligence applications, with OpenAI's ChatGPT being one of the most popular tools. This literature review explores the trends in the application of ChatGPT in library settings, focusing on user engagement and support services from 2015 to 2023.
• ChatGPT as a Tool for Library Research – Some Notes and Suggestions I see ChatGPT and its alternatives as having partial value as tools for library searching. You can use them without any training, but they will perform better when you know some details about them.

9 Ways To Use ChatGPT To Write A Literature Review (WITHOUT Plagiarism) Video

• Step-by-Step

How to Use ChatGPT to Write a Literature Review With Prompts

Dr. Somasundaram R | https://www.ilovephd.com/ | Copyright © 2019-2023 – iLovePhD | May 19, 2023

Writing a literature review can be a challenging task for researchers and students alike. It requires a comprehensive understanding of the existing body of research on a particular topic. However, with the advent of advanced language models like ChatGPT, the process has become more accessible and efficient.

Discover how to effectively utilize ChatGPT as a research assistant to write a comprehensive and SEO-friendly literature review. Follow our step-by-step guide to leverage this powerful tool, optimize your review for search engines, and contribute to the scholarly conversation in your field.

A Step-by-Step Guide: How to Use ChatGPT for Writing a Literature Review

Step 1: Defining Your Research Objective Before diving into the literature review process, it is crucial to define your research objective.

Clearly articulate the topic, research question, or hypothesis you aim to address through your literature review. This step will help you maintain focus and guide your search for relevant sources.

Step 2: Identifying Keywords and Search Terms To effectively use ChatGPT to assist in your literature review, you need to identify relevant keywords and search terms related to your research topic.

These keywords will help you narrow down your search and gather pertinent information. Consider using tools like Google Keyword Planner or other keyword research tools to discover commonly used terms in your field.

Step 3: Familiarizing Yourself with ChatGPT Before engaging with ChatGPT, it is essential to understand its capabilities and limitations. Familiarize yourself with the prompts and commands that work best with the model.

Keep in mind that ChatGPT is an AI language model trained on a vast amount of data, so it can provide valuable insights and suggestions, but it’s important to critically evaluate and validate the information it generates.

Step 4: Generating an Initial Literature Review Outline Start by creating an outline for your literature review. Outline the main sections, such as the introduction, methodology, results, discussion, and conclusion.

Within each section, jot down the key points or subtopics you want to cover. This will help you organize your thoughts and structure your review effectively.

Step 5: Engaging with ChatGPT for Research Assistance Once you have your outline ready, engage with ChatGPT for research assistance.

Begin by providing a clear and concise prompt that specifies the topic, context, and any specific questions you have. For example, “What are the current trends in [your research topic]?” or “Can you provide an overview of the main theories on [your research question]?”

Step 6: Reviewing and Selecting Generated Content ChatGPT will generate a response based on your prompt. Carefully review the content generated, considering its relevance, accuracy, and coherence.

Extract key points, relevant references, and insightful arguments from the response and incorporate them into your literature review. Be sure to cite and attribute the sources appropriately.

Step 7: Ensuring Coherence and Flow While ChatGPT can provide valuable content, it’s important to ensure the coherence and flow of your literature review.

Use your critical thinking skills to connect the generated content with your research objective and existing knowledge. Rearrange, rephrase, and expand upon the generated text to ensure it aligns with the structure and purpose of your review.

Step 8: Editing and Proofreading Once you have incorporated the generated content into your literature review, thoroughly edit and proofread the document.

Check for grammatical errors, consistency in referencing, and overall clarity. This step is crucial to ensure your literature review is polished and professional.

ChatGPT prompts to Write a Literature Review

Prompts you can use when engaging with ChatGPT for research assistance in writing a literature review:

“Can you provide an overview of the main theories and concepts related to [your research topic]?”

“What are the current trends and developments in [your research field]?”

“Can you suggest some key studies or research papers on [specific aspect of your research topic]?”

“What are the main methodologies used in conducting research on [your research topic]?”

“Can you provide a critical analysis of the existing literature on [your research question]?”

“Are there any gaps or areas of controversy in the literature on [your research topic] that need further exploration?”

“What are the key findings and conclusions from the most recent studies on [your research topic]?”

“Can you suggest some reputable journals or publications explore for relevant literature in [your research field]?”

“What are the different perspectives or schools of thought in the literature on [your research topic]?”

“Can you provide a summary of the historical background and evolution of research on [your research topic]?”

Remember to provide clear and specific instructions in your prompts to guide ChatGPT in generating relevant and accurate content for your literature review.

Using ChatGPT to write a literature review can greatly facilitate the research process. By following a step-by-step approach, researchers can effectively leverage ChatGPT’s capabilities to gather insights, generate content, and enhance the quality of their literature review. However, it is important to approach the generated content critically, validate it with reliable sources, and ensure coherence within the review.

• << Previous: Art + Image Prompts
• Next: Prompt Literacy >>
• Last Updated: May 17, 2024 11:00 AM
• URL: https://library.fiu.edu/ai

80+ ChatGPT Prompts for Literature Review

Conducting a literature review can often seem like a daunting task. It's an essential part of academic and research work, involving comprehensive and critical analysis of previously published research related to a specific topic. The goal is to gain a deep understanding of the state of knowledge on the topic, identify gaps, and provide a foundation for further research.

Traditionally, literature reviews have been manual, time-consuming processes. However, the advancement of technology, especially Artificial Intelligence (AI), is reshaping this landscape. One such development is ChatGPT, an AI developed by OpenAI. This tool can assist researchers in conducting more efficient and thorough literature reviews.

In this blog post, we will delve into the ways that ChatGPT, with its remarkable language understanding and generation capabilities, can be used to streamline your literature review process.

From explaining the basic capabilities of ChatGPT to providing essential prompts and tips to conduct a literature review, this post will serve as a comprehensive guide for those looking to leverage AI for academic research.

Stay tuned as we navigate this fascinating convergence of AI and research, and explore the potential of ChatGPT in making literature reviews more efficient and effective.

Table of Contents

The Power of AI in Literature Reviews

The role of Artificial Intelligence (AI) in academia and research is expanding rapidly. AI has emerged as a game-changing tool that can significantly enhance the process of conducting literature reviews.

The Benefits of AI in Conducting Literature Reviews

There are numerous advantages to using AI for literature reviews. Let's delve into these benefits:

Time Efficiency

AI can streamline the literature review process, significantly reducing the time taken. With AI, you can quickly scan vast amounts of literature, saving valuable time that can be used for analysis and synthesis.

Comprehensive Coverage

AI algorithms are capable of scanning and analyzing a far greater volume of literature than a human could reasonably manage. This ensures a comprehensive and exhaustive review of existing literature.

Consistency and Bias Reduction

AI maintains a level of consistency that is hard to achieve manually. It does not suffer from fatigue or bias, ensuring that all relevant literature is considered fairly.

The Role of ChatGPT in Literature Reviews

ChatGPT, developed by OpenAI, has proven to be a valuable tool for conducting literature reviews. It is a language model that can understand and generate human-like text, making it an excellent tool for understanding complex academic literature and providing summaries or insights.

Here are some example prompts that can be used to guide ChatGPT in assisting with literature reviews:

Example Prompts:

Generate a summary of this research paper.

What are the key findings of this study?

How does this research contribute to the field of [specific field]?

Identify the strengths and weaknesses of this study.

Compare this study with the previous study on the same topic.

What are the research gaps identified in this literature?

What future research directions are suggested by this study?

Preparing for a ChatGPT-Assisted Literature Review

Before diving into a literature review with ChatGPT, you need to set the stage right. Preparation is crucial to ensuring you get the most out of your AI-assisted literature review process.

Detailed Steps on How to Set Up for a Literature Review Using ChatGPT

• Understand Your Topic : Before starting, ensure you have a clear understanding of your topic and the specific aspects you want to focus on in the literature review.
• Gather Your Materials : Collect all the papers, articles, and sources you plan to include in your literature review.
• Organize Your Materials : Categorize your sources based on various parameters like theme, methodology, results, etc. This will make it easier to navigate through them during the review.
• Prepare Your Prompts : Create a list of prompts you plan to use with ChatGPT. These prompts should be designed to extract the information you need from each source.

Tips for Effective Use of ChatGPT in Literature Review

Preparing a list of keywords or topics.

Start with a broad overview of your research topic and gradually narrow down the scope by identifying specific sub-topics, methodologies, or research questions you want to explore. Use these as keywords for your prompts.

Having Clear Objectives for the Review

Before beginning your review, clearly outline what you hope to achieve. This could be understanding the current state of research, identifying gaps, comparing methodologies, etc.

Understanding the Limits of AI Assistance

While ChatGPT is a powerful tool, it's essential to remember it is a tool and not a replacement for human analysis. Use it to process and summarize information, but the interpretation and synthesis of this information should be done by you.

Provide an overview of the literature on [specific topic].

Summarize the methodologies used in the studies on [specific topic].

Identify common themes in the literature on [specific topic].

What are the disagreements in the literature about [specific topic]?

Analyze the impact of [specific study] on the broader field of [specific topic].

What are the emerging trends in the literature on [specific topic]?

What are the commonly cited sources in the literature on [specific topic]?

Essential ChatGPT Prompts for Literature Review

ChatGPT is a powerful tool for conducting literature reviews, and the key to using it effectively is crafting the right prompts. These prompts guide the AI in scanning, analyzing, and summarizing the literature based on your specific needs.

Need To Write A Thesis? ➡️ 87+ ChatGPT Prompts for Thesis Writing

Initial Exploration of a Topic

When you're just starting a literature review, you'll need prompts that help you gain a broad understanding of your research topic. These prompts can assist in grasping the general landscape of the literature.

Give a brief overview of research done on [topic].

What are the prominent theories or models in [topic]?

What are the key debates or controversies in [topic]?

Who are the key authors or researchers in [topic]?

What methodologies are commonly used in studies on [topic]?

Summarize the historical development of [topic] in academia.

What have been the significant shifts or changes in [topic] over time?

Deep Diving into Specific Studies

Once you have a broad understanding, you'll want to delve into specific studies or papers. These prompts help you extract detailed information from individual sources.

Summarize the main arguments and findings of [specific paper].

What methodology does [specific paper] use and why?

What are the strengths and weaknesses of [specific paper]?

How does [specific paper] contribute to the broader field of [topic]?

Compare the findings of [specific paper] with [another specific paper].

What future research does [specific paper] suggest or inspire?

How has [specific paper] been received or critiqued by other scholars in the field?

Comparing and Contrasting Different Works

ChatGPT can also help you identify similarities and differences between various sources. These prompts are useful for synthesizing information across multiple works.

Compare and contrast the methodologies of [paper 1] and [paper 2].

How do the findings of [paper 1] and [paper 2] agree or disagree?

How does [author 1]'s perspective on [topic] differ from [author 2]'s?

Compare the theoretical frameworks used in [paper 1] and [paper 2].

How does the approach to [topic] change over time as seen in [earlier paper] vs. [later paper]?

What themes are common across these studies on [topic]?

What disparities or contradictions exist in the literature on [topic]?

Summarizing Findings

Finally, as you conclude your literature review, you'll want to summarize your findings. These prompts help you consolidate your insights and present them clearly.

Summarize the key findings from my literature review on [topic].

What trends or patterns have emerged from the literature on [topic]?

What gaps or unanswered questions exist in the literature on [topic]?

How have perspectives on [topic] evolved over time, based on the literature?

What consensus, if any, exists in the literature on [topic]?

How does the literature inform our understanding of [topic]?

What future research directions are suggested by the literature on [topic]?

How to Create Effective ChatGPT Prompts

Crafting effective prompts for ChatGPT is part art, part science. It involves understanding the workings of the AI model, your research objectives, and the specific information you seek from your literature review.

Guidelines for Creating Effective ChatGPT Prompts

Here are some general guidelines to follow when crafting prompts for ChatGPT:

• Be Specific : The more specific your prompt, the more focused and relevant the AI's response will be.
• Use Clear Language : Make your prompts as clear and concise as possible. Avoid ambiguous terms and complex phrasing.
• Guide the AI : To get the best results, guide the AI towards the kind of response you're looking for. For example, if you want a summary, explicitly ask for a summary.
• Experiment and Iterate : Different prompts can yield different results. Don't be afraid to tweak your prompts and try different approaches to find what works best.

Tips for Refining and Iterating on Prompts

Prompts may not always work perfectly on the first try. Here are some tips for refining and iterating on your prompts:

• Analyze the Responses : Look at the AI's responses to understand how it is interpreting your prompts. If the responses are not what you expected, tweak your prompt to guide the AI more accurately.
• Try Different Angles : If a prompt isn't working, try approaching the question from a different angle or rephrasing the prompt.
• Split Complex Prompts : If a prompt is too complex, try splitting it into multiple simpler prompts.

Provide a summary of the main arguments in [specific paper].

What are the primary findings of [specific paper]?

Analyze the methodology used in [specific paper].

What are the implications of [specific paper] for future research?

Compare the findings of [specific paper] with those of [another specific paper].

What critiques have been made about [specific paper]?

What are the key themes across the literature on [specific topic]?

Examples of Good and Less Effective Prompts

Here are some examples of good and less effective prompts to help guide your prompt crafting:

Good Prompts:

Less Effective Prompts:

• Tell me about [specific paper]. (This is too vague and doesn't guide the AI towards the specific information you're seeking.)
• What's interesting about [specific paper]? (This prompt is subjective and may not yield useful information for a literature review.)
• What's everyone saying about [topic]? (This prompt is overly broad and might result in a wide range of responses that aren't focused enough for a literature review.)

Navigating Challenges in Using ChatGPT for Literature Reviews

While ChatGPT can be a powerful tool for conducting literature reviews, it's important to understand its limitations and potential challenges. Being aware of these can help you navigate the process more effectively.

Understanding the Limitations of ChatGPT

Despite its sophisticated capabilities, ChatGPT is not without its limitations. Here are some to keep in mind:

• Dependency on Input Quality : The quality of ChatGPT's responses depends on the quality and clarity of the prompts you provide.
• Lack of Critical Analysis : While ChatGPT can summarize and synthesize information, it can't provide the same level of critical analysis and judgment that a human researcher can.
• Potential for Misinterpretation : ChatGPT may occasionally misinterpret prompts or generate responses that miss the mark.

Overcoming Challenges in Using ChatGPT for Literature Reviews

Despite these limitations, there are ways to overcome the challenges in using ChatGPT for literature reviews:

• Carefully Craft Your Prompts : As noted earlier, the quality of your prompts greatly influences the AI's output. Spend time crafting and refining your prompts.
• Double-Check the Information : Always cross-check the AI's responses against the original sources to ensure accuracy.
• Combine AI and Human Analysis : Use ChatGPT as a tool to assist with your literature review, not as a replacement for your own analysis.

Here are some example prompts that can help you navigate potential challenges:

If the response seems off: Could you provide another summary of [specific paper]?

For double-checking information: What are the key findings of [specific paper]?

To ensure comprehensive coverage: Are there any other significant points in [specific paper] that haven't been mentioned?

If the response is too broad: Could you provide a more focused summary of the methodology in [specific paper]?

If the response is too narrow: What are the broader implications of [specific paper] for the field?

To clarify a complex point: Can you explain the main argument in [specific paper] in simpler terms?

To guide the AI's interpretation: Based on [specific paper], what are the potential areas for future research in this field?

More Great ChatGPT Prompts for Literature Reviews

An engaging literature review requires delving into a range of perspectives, methodologies, and findings within your chosen area of study. Crafting prompts that encourage varied and in-depth analysis can help you gain a comprehensive understanding of your topic. Here are more unique prompt ideas to enrich your literature review process using ChatGPT:

What are the key variables studied in relation to [topic]?

What theoretical frameworks have been applied to the study of [topic]?

Explain the significance of [specific study] in the field of [topic].

What are the ethical considerations discussed in the literature on [topic]?

How does [specific paper] challenge the prevailing understanding of [topic]?

What statistical methods are commonly used in studies on [topic]?

Discuss the influence of [specific author] on the research of [topic].

How have the research methods evolved over time in the study of [topic]?

What are the socio-cultural considerations in the literature on [topic]?

How does [specific paper] address the limitations of previous studies on [topic]?

Describe the study design of [specific paper] in the context of [topic].

What contributions does [specific paper] make to the existing literature on [topic]?

How have policy changes affected the research on [topic] over time?

What is the role of technology in the research of [topic]?

Identify the key longitudinal studies in the field of [topic].

What are the demographic considerations in studies on [topic]?

How does [specific paper] incorporate interdisciplinary approaches to [topic]?

What biases or assumptions are present in the literature on [topic]?

How does [specific paper] use empirical data to support its arguments on [topic]?

Discuss the replication studies in the field of [topic].

What are the implications of the literature on [topic] for practitioners?

How does the literature on [topic] align or contrast with public opinion?

What are the critical reviews or meta-analyses in the field of [topic]?

How does [specific paper] discuss the limitations of its own study on [topic]?

Identify the seminal works in the field of [topic] and explain their significance.

In the rapidly evolving world of AI, tools like ChatGPT have the potential to revolutionize the way we approach tasks such as literature reviews. This powerful AI model, when provided with carefully crafted prompts, can sift through and synthesize vast amounts of information, assisting you in navigating the complex landscape of academic literature.

Remember, the power of ChatGPT lies in its ability to respond to diverse prompts. While this blog post has offered several useful prompts and tips for conducting a literature review, don't hesitate to experiment and innovate with your own prompts. Your creativity and critical thinking, combined with ChatGPT's capabilities, can turn the daunting task of literature review into a more engaging and manageable process.

However, it's important to bear in mind the limitations of AI and to always complement AI-generated insights with your own critical analysis.

As you embark on your journey of exploration and discovery in your chosen field of study, may ChatGPT prove a valuable ally in helping you uncover the rich tapestry of ideas, debates, and breakthroughs that form the essence of academic literature.

75+ ChatGPT Prompts for Presentations

120+ ChatGPT Prompts for Social Media Posts

About R.J. Adams

R.J. is an artificial intelligence aficionado, committed to demystifying the intricacies of this cutting-edge technology for enthusiasts and novices alike. Leveraging his comprehensive background in internet marketing, online tools, and communications, he continually ventures into the landscape of newly released AI tools, conducting thorough tests and analyses, as well as finding the best prompts to use on the latest AI tools. He can also solve Rubik's Cube in under 2 minutes.

Connect With Me:

Reach out to us for sponsorship opportunities..

We look forward to talking to you!

© 2024 AIHabit.net

Literature Reviews

• Getting Started
• Choosing a Type of Review
• Developing a Research Question
• Searching the Literature
• Searching Tips
• ChatGPT [beta]
• ChatGPT Resources
• Documenting your Search
• Using Citation Managers
• Concept Mapping
• Writing the Review
• Further Resources

UM's Report on GenerativeAI

The initial report from the U-M Generative Artificial Intelligence Advisory (GAIA) Committee.

—User-friendly Interface that allows faculty, staff and students to engage in chat-based queries and benefit from the expertise of GENAI technology

— U-M GPT, enhances, teaching learning research in collaboration, providing a valuable and equitable resource for the entire university community

— designed to be accessible, including for use with screen readers

— provides access to GPT-3.5 and GPT-4 with no charge

https://chat.openai.com

• The knowledge cutoff for the ChatGPT 3.5 is September 2021 and it has no access to the Internet. Academic users may consider alternatives such as  Semantic Scholar,  Elicit,  Consensus  or other AI-powered tools for doing  real-time  literature reviews.
• Input/ Output length  - ChatGPT-3.5 allows a maximum token limit of 4096 tokens. According to ChatGPT " On average, a token in English is roughly equivalent to 4 bytes or characters. English words are typically around 5 characters long. This means that, very roughly, you could fit around 800 to 1000 English words within 4096 tokens." 
• Notes : (1) According to ChatGPT, the generated responses are non-deterministic by default. So if you run the searches again and get slightly or very different results than the ones listed on this guide, it's likely due to this factor. (2) While ChatGPT or Bing Chat searches offer conversational interaction, they might not be a substitute for the rigor and reliability of traditional academic database searches.  

KEYWORD GENERATION

Write a literature review abstract based on the following keywords: Keywords: Artificial Intelligence, Employment Trends, Skill Development, Future Workforce

Infant death Neonatal mortality  (referring specifically to deaths within the first 28 days of life) Perinatal mortality  (referring to deaths occurring during the period from 22 completed weeks of gestation to 7 completed days after birth) Early childhood mortality  (referring to deaths occurring within the first five years of life) Child mortality  (referring to deaths occurring before the age of 18) [Press the  Regenerate  button to get more.]

• Example -   RELATED WORDS What are the related words of infant mortality?

Neonatal mortality, Post-neonatal mortality, Stillbirths, Low birth weight, Malnutrition, Infectious diseases, Vaccination, Maternal health,  Access to healthcare, Poverty, Social inequality, Sanitation, Hygiene, Water quality, Childbirth complications, Congenital abnormalities, Birth defects, Maternal age, Under-five mortality, Child mortality, Perinatal mortality, Preterm birth, Low birth weight, Sudden infant death syndrome (SIDS), Maternal mortality, Postnatal care, Malnutrition, Immunization, Access to healthcare, Clean water and sanitation, Congenital anomalies, Infant health disparities, Infant mortality rate (IMR), Infant survival.  

(("Infant Death"[Mesh] OR "Sudden Infant Death Syndrome"[Mesh] OR "Infant Mortality"[Mesh] OR "Perinatal Mortality"[Mesh] OR "Neonatal Mortality"[Mesh] OR "Fetal Mortality"[Mesh] OR "Infant Death" OR "Sudden Infant Death Syndrome" OR "Infant Mortality" OR "Perinatal Mortality" OR "Neonatal Mortality" OR "Fetal Mortality") AND ("Risk Factors"[Mesh] OR "Etiology"[Mesh] OR "Prevention and Control"[Mesh] OR "Interventions"[Mesh] OR "Public Health"[Mesh] OR "Health Disparities"[Mesh] OR "Healthcare Disparities"[Mesh] OR "Poverty"[Mesh] OR "Malnutrition"[Mesh] OR "Infection"[Mesh] OR "Congenital Abnormalities"[Mesh] OR "Prematurity"[Mesh] OR "Low Birth Weight"[Mesh] OR "Maternal Education"[Mesh] OR "Maternal Health"[Mesh] OR "Risk Factors" OR "Etiology" OR "Prevention and Control" OR "Interventions" OR "Public Health" OR "Health Disparities" OR "Healthcare Disparities" OR "Poverty" OR "Malnutrition" OR "Infection" OR "Congenital Abnormalities" OR "Prematurity" OR "Low Birth Weight" OR "Maternal Education" OR "Maternal Health"))

Comments and Suggestions: #1 -This search string may not work on all database platforms. For example,  Medline (OVID)  may not be able to run it,  Medline  &  CINAHL (EBSCO)  can run it without any problems while   PubMed  will run it but it will indicate which MeSH terms are not indexed. See PubMed's search  History and Details  under  Advanced . #2 -This is just one possible search string and you may need to adjust it based on your specific research questions and the databases you are using. #3 - It might be easier if you break the task into 3 parts: ask the chatbot give you the synonyms > ask it to give you the thesauri terms > and ask it to combine them.

RESEARCH QUESTION

Increased risk of sudden infant death syndrome (SIDS) Increased risk of premature birth Increased risk of low birth weight Increased risk of respiratory problems in newborns Increased risk of respiratory problems in infants exposed to secondhand smoke Increased risk of developing asthma and other respiratory illnesses later in life for infants exposed to secondhand smoke Please generate a literature review that addresses the research question: 'What are the effects of social media on the mental health of adolescents? [Author's note : If you ask for the specific studies, ChatGPT may generate   non-existent references or false knowledge. To find out   why Large Language Models   hallucinate, check out this WiKi article:  Hallucination (artificial intelligence)   and this blog post -   A Gentle Introduction to Hallucinations in Large Language Models  by Adrian Tam ]

DATABASE RECOMMENDATIONS  

Heath Science  -  Which databases to use to find articles on the EFFECTS OF MATERNAL SMOKING ON INFANT DEATH? PubMed:  This is a database of biomedical literature from the National Library of Medicine. It includes articles from medical journals and other sources on a wide range of health-related topics. Embase:  This is a database of biomedical literature that covers more international and pharmaceutical content than PubMed. It includes articles from medical journals and other sources on a wide range of health-related topics. Cochrane Library : This is a collection of databases that provides systematic reviews and meta-analyses of healthcare interventions, including interventions related to maternal smoking and infant death. Scopus:  This is a multidisciplinary database that covers research from a wide range of fields, including health sciences. It includes articles from scientific journals, books, and conference proceedings. Web of Science:  This is a database that covers research from a wide range of fields, including health sciences. It includes articles from scientific journals, books, and conference proceedings. CINAHL:  This is a database of nursing and allied health literature, including articles on maternal smoking and infant death. [Note: The response above was copied and pasted directly from ChatGPT. No editing was added.] Education  --  I am doing a  meta-analysis  on k-12 STEM education. What databases should I use? ERIC : This is a database of education research and information, including articles from journals, reports, and conference papers. It is a commonly used database for educational research. Education Source : This is a comprehensive database of education research and information, including articles from journals, books, and conference papers. Web of Science:  This is a multidisciplinary database that covers research from a wide range of fields, including education. It includes articles from scientific journals, books, and conference proceedings. Scopus:  This is another multidisciplinary database that covers research from a wide range of fields, including education. It includes articles from scientific journals, books, and conference proceedings. PsycINFO:  This is a database of psychology research, including articles from journals, books, and dissertations. It can be useful for research on STEM education that includes psychological factors. Cochrane Library : This is a collection of databases that provides systematic reviews and meta-analyses of healthcare interventions including interventions related to education.

Example:   ACT-AS-AN-EXPERT PROMPT Here's an alternative way to consolidate all the questions into a single consultation by using this popular Act-As-An-Expert prompt, aka, system message prompt:  I am going to conduct a systematic review on games and mathematics related to K-12 education  I want you to act as a systematic review expert. I will ask you some questions related to my research topic and you advise me on how to do it. 

GREY LITERATURE  

Please show me all the resources that offer grey literature for conducting research in the field of education     ERIC (Education Resources Information Center): https://eric.ed.gov/     OpenGrey: http://www.opengrey.eu/     British Education Index: https://www.ebscohost.com/academic/british-education-index     WorldCat: https://www.worldcat.org/     ProQuest Dissertations & Theses Global: https://www.proquest.com/products-services/pqdtglobal.html     EThOS (Electronic Theses Online Service): https://ethos.bl.uk/     Institutional Repositories of universities and academic institutions (search for specific institution repositories)     American Educational Research Association (AERA): https://www.aera.net/     National Education Association (NEA): https://www.nea.org/     Association for Supervision and Curriculum Development (ASCD): http://www.ascd.org/     RAND Corporation: https://www.rand.org/     National Center for Education Research (NCER): https://ies.ed.gov/ncer/     National Center for Education Statistics (NCES): https://nces.ed.gov/     European Educational Research Association (EERA): https://www.eera-ecer.de/     UNESCO International Institute for Educational Planning (IIEP): http://www.iiep.unesco.org/     Organization for Economic Cooperation and Development (OECD): https://www.oecd.org/education/     Education Development Center (EDC): https://www.edc.org/     National Institute for Educational Policy Research (NIER) - Japan: https://www.nier.go.jp/     Australian Council for Educational Research (ACER): https://www.acer.org/     Canadian Education Association (CEA): https://www.cea-ace.ca/ My topic is about stem education. Which grey literature databases should I use?    STEM Education Research Centre (SERC) Grey Literature Database    National STEM Education Research Repository (NSERR)    National Science Digital Library (NSDL)    STEM Education Resource Center (SERC) GrayLit Network    STEM Education Grey Literature Library

MORE RESOURCES

• << Previous: AI Lit Searching [beta]
• Next: Organizing the Literature >>
• Last Updated: May 9, 2024 11:44 AM
• URL: https://guides.lib.umich.edu/litreview

• Joyner Library
• Laupus Health Sciences Library
• Music Library
• Digital Collections
• Special Collections
• North Carolina Collection
• Teaching Resources
• The ScholarShip Institutional Repository
• Country Doctor Museum

Literature Review: Using ChatGPT

• About Literature Reviews
• How to Conduct your Lit Review
• A Few Writing Tips
• A Few Helpful Resources
• A Silly Example
• Using ChatGPT

What is AI, ChatGPT, and large-language modeling?

Artificial intelligence (AI) , in the context of this guide, is the use of datasets to train computer models to emulate human thought processes (IBM, n.d.). It is the emulation itself that may convince information consumers of the supposed "intelligence" of a machine; the more convincing the emulation, the more users are prone to trust the generated information.

ChatGPT is a generative large language model which has gained popularity since late 2021. The name stands for Chat Generative Pre-trained Transformer. GPT models are programmed to mimic the look and feel of engaging in a realistic conversation with another human being; however, bias and inaccuracy remain a considerable risk ( Kocoń, J., 2023).

Large language models (LLMs) are AI models that utilize networked datasets, emulating the neural pathways of the human brain, which assists in the appearance of actual cognition (Birhane et al., 2023).

Birhane, A., Kasirzadeh, A., Leslie, D., & Wachter, S. (2023). Science in the age of large language models. Nature Reviews Physics . https://www.nature.com/articles/s42254-023-00581-4

IBM. (n.d.). What is artificial intelligence (AI)? https://www.ibm.com/topics/artificial-intelligence Retrieved 27 April 2023.

Read more here: Generative AI in the Classroom & Research

What can I use ChatGPT for?

First and most importantly: check with your instructor that using generative AI is allowed for your class! This information should be added in your course syllabus with language indicating whether and to what extent AI can be used on assignments. If you are not sure, ask your instructor first .

From my biased perspective (that of a librarian), here is a brief list of ways you could use ChatGPT (if allowed for your class). This list is not exhaustive of all ideas.

• Keyword generation. Having trouble thinking of other ways to describe a concept? Ask ChatGPT to help!
• Narrowing/broadening your topic. Not getting any results? Getting way too many results? Use ChatGPT to help you brainstorm ways you can adjust your research question so that you can find the most effective sources.
• Setting the right tone. Want to create a patient education document in plain language but not quite sure how to translate the nursing jargon into easy-to-understand statements? ChatGPT can help you think of ways to ensure your tone matches your preferred audience.

What shouldn't I use ChatGPT for?

Because of bias and limitations from human contributions, it is imperative to approach generative AI with caution.

If your instructor has approved the use of ChatGPT or other generative AI in the class, below is a brief list of ways you should NOT use it. Again, this list is not exhaustive.

• Writing your assignments. This may be considered a form of plagiarism as the generated language is not your own original writing. If you use ChatGPT in this way without credit, you may be found in violation of the University's Academic Integrity policies. If you aren't sure, check with your instructor.
• Searching for sources. Do not use ChatGPT in lieu of conducting a literature search through the library resources. ChatGPT has a bad habit of "hallucinating" results; in other words, it will generate answers that sound right but that are not actual sources that exist. ChatGPT is not a search engine or a database.
• Creating accurate citations. ChatGPT is about as good as any other citation generator out there, which is to say that it is hit or miss. ChatGPT may leave off required elements of a citation, invent elements (for instance, generating incorrect DOIs or URLs), and fail to follow citation style formatting requirements. If you use ChatGPT for citation generation, be sure to double- and triple-check every citation thoroughly before submitting your assignment.

Ask a Librarian

Chat with a librarian is available 9-5 Monday-Saturday.

You can also  contact a librarian .

1-888-820-0522 (toll free)

252-744-2230

Or email us:

[email protected]

Provide Feedback on this Guide

Please follow the link below to complete a short survey and provide feedback on this guide.

Research Guide Satisfaction Survey

Should I cite ChatGPT?

Citing ChatGPT and other generative AI will allow your work to remain transparent while also demonstrating ethical considerations. If generative AI is used in research, best practice would dictate to mention the use of this tool in the narrative.

Suggestions for citations may be found below:

• American Psychological Association (APA)
• Chicago Manual of Style
• Modern Language Association (MLA)
• << Previous: A Silly Example
• Last Updated: Jan 12, 2024 1:21 PM
• URL: https://libguides.ecu.edu/lit_reviews

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Am J Cancer Res
• v.13(4); 2023
• PMC10164801

The role of ChatGPT in scientific communication: writing better scientific review articles

Jingshan huang.

1 School of Computing and College of Medicine, University of South Alabama, Mobile, AL, USA

2 Institute of Biochemistry and Molecular Biology, Institute of Biomedical Sciences, and Research Center for Cancer Biology, China Medical University, Taichung, Taiwan

Artificial intelligence tools represent an exciting opportunity for scientists to streamline their research and write impactful articles. Using artificial intelligence tools like ChatGPT can greatly improve writing review articles for scientists, by enhancing efficiency and quality. ChatGPT speeds up writing, develops outlines, adds details, and helps improve writing style. However, ChatGPT’s limitations must be kept in mind, and generated text must be reviewed and edited to avoid plagiarism and fabrication. Despite these limitations, ChatGPT is a powerful tool that allows scientists to focus on analyzing and interpreting literature reviews. Embracing these tools can help scientists produce meaningful research in a more efficient and effective manner, however caution must be taken and unchecked use of ChatGPT in writing should be avoided.

Introduction

The use of artificial intelligence (AI) tools such as ChatGPT (OpenAI, San Francisco, CA), is becoming increasingly important in scientific writing [ 1 - 5 ]. Whether you like or hate it, you need to face the fact that many other people are using ChatGPT to generate a lot of manuscripts right now [ 6 - 9 ]. Instead of resisting it or wasting your time to blame it, a better choice is for you to manage to use this powerful tool as your personal assistant, ethically, to increase your productivity and the quality of your works.

Using ChatGPT is a powerful tool to help scientists to write review articles more efficiently. Here are several reasons why you should use it to increase your proficiency in review writing, speed up your writing process, and save time [ 10 - 12 ].

It saves us time. Writing a review article can be a time-consuming process, involving extensive research, organization, and writing. AI tools like ChatGPT can speed up the writing process by automatically generating contents that can be further edited and refined by human authors, saving valuable time and resources.

It can help us to manage your data. Scientists often need to sift through large volumes of data and research papers to find relevant information for their review articles. AI tools like ChatGPT can assist with data management by analyzing and summarizing large amounts of information effectively and much more quickly than humans.

It can help us to improve the quality of our scientific writing. AI tools like ChatGPT can help to improve the quality of our writing by identifying potential errors, inconsistencies, or gaps in the argument. This can help human authors to refine their writing and ensure that their manuscript is accurate, well-structured, and well-supported by all available evidence as much as possible.

It can help us to keep a more balanced perspective. ChatGPT is not personally biased and can thus provide diverse perspectives on a given topic. This can be particularly useful in fields where there are many conflicting opinions and viewpoints, for example, promoting a ketogenetic diet for patients with the cardiovascular disease.

Moreover, if you are not a native English speaker, ChatGPT can be tremendously helpful. Are you tired of being criticized by the reviewers and editors on your English writings for not using the standard English, and suggest you to ask a native English speaker to help proofreading or even use the service from a professional English editor? ChatGPT can readily help you with grammar and sentence structure, suggest appropriate vocabulary choices, assist in translating text from one language to another, so on and so forth.

In addition to the above reasons, ChatGPT can help suggesting the article title, shortening, or expanding the abstract, discussing the results, and even recommending creative ideas.

ChatGPT can also help us to do the plagiarism detection. AI tools have been utilized to assist with detecting plagiarism by checking the similarity of the text with existing, published sources, helping to ensure that the manuscript is original and not copied from others’ work.

Overall, the use of AI tools like ChatGPT can significantly help scientists to write review articles more efficiently and accurately, thus improving the quality and impact of their research outcomes.

How ChatGPT works

ChatGPT is an AI language model developed by OpenAI [ 1 , 2 ]. It is based on a deep neural network architecture called the transformer model, which adopts the mechanism of self-attention, differentially weighting the significance of each part of the input data. The transformer model is pretrained on an extremely large corpus of text data to learn context and thus meaning by tracking relationships in sequential data such as the words in a sentence. After the pretrained phase, the transformer model is then able to generate natural language responses to user inputs.

When a user inputs a text prompt, ChatGPT generates a response based on its understanding of the input and the patterns it has previously learned from the text data it has been trained on. ChatGPT is capable of generating responses in a variety of formats, including short answers, long-form essays, and even conversation-style interactions [ 1 , 2 ].

ChatGPT works by breaking down the input text into a sequence of tokens, which are then processed by the transformer model to generate a probability distribution over the next token in the sequence. The model then selects the token with the highest probability and generates the corresponding output text. This process is conducted iteratively, with the model generating a new token and output text at each step, until a stopping criterion is met [ 1 , 2 ].

AI tools like ChatGPT is capable of generating responses in a wide range of domains, including scientific writing, creative writing, and general conversation. Its ability to generate natural language responses that are coherent and contextually relevant has made it a popular tool for a wide range of applications, including automated content creation, language translation, and natural language processing, just to name a few [ 1 - 5 , 13 - 16 ].

How ChatGPT can assist scientists in writing

Chatgpt can assist scientists in conducting literature reviews.

Topic selection: ChatGPT can help scientists to select a suitable topic for their literature review by generating relevant keywords and suggesting related and meaningful research areas. For example, a biologist could input “What are the latest research areas in the field of molecular biology?” and ChatGPT could generate a list of relevant keywords and research areas, such as “CRISPR-Cas9 gene editing”, “single-cell sequencing”, and “epigenetics”.

Literature search: ChatGPT can assist scientists in conducting literature searches by generating relevant search queries and suggesting relevant databases and resources. For example, a biologist could input “What are the best databases to search for articles on CRISPR-Cas9 gene editing?” and ChatGPT could generate a list of relevant databases and search queries, such as “PubMed”, “CRISPR-Cas9 AND gene editing”, and “CRISPR-Cas9 AND therapeutic applications”.

Article selection: ChatGPT can help scientists to select relevant articles for their literature review by generating summaries and providing context for each article. For example, a biologist could input “Can you summarize the findings of the latest review article on single-cell sequencing in cancer research?” and ChatGPT could generate a summary of the article, highlighting its key findings and relevance to the topic at hand.

Citation and referencing: ChatGPT can assist scientists in accurately citing and referencing their sources by generating the appropriate citation format and suggesting related articles to cite. For example, a biologist could input “How do I cite a journal article using the APA citation style?” and ChatGPT could generate the appropriate citation format and provide examples of related articles to cite.

Overall, ChatGPT can assist scientists in conducting literature reviews by helping them to select relevant topics, conduct literature searches, select articles, and accurately cite and reference their sources. By automating many of the time-consuming and tedious tasks associated with literature reviews, ChatGPT can help scientists to conduct more comprehensive and efficient reviews, leading to higher quality review manuscripts, and, in a much more efficient manner.

ChatGPT can assist scientists in developing outlines

Inputting the topic: Scientists can input the topic of their review article, for example, “The role of epigenetics in cancer development and treatment”.

Generating subtopics: ChatGPT can generate a list of subtopics related to the main topic. For example, ChatGPT could suggest subtopics such as “Epigenetic modifications in cancer cells”, “Epigenetic therapy for cancer”, and “Epigenetic biomarkers for cancer diagnosis”.

Organizing subtopics: ChatGPT can help scientists to better organize the subtopics into a logical outline for their review article. For example, ChatGPT could suggest organizing the subtopics under main headings such as “Introduction”, “Epigenetic modifications in cancer”, “Epigenetic therapy for cancer”, and “Epigenetic biomarkers for cancer diagnosis”.

ChatGPT can assist scientists in adding details

ChatGPT can assist scientists in adding greater details to the outline by suggesting key points and relevant literature for each subtopic. For example, ChatGPT could suggest adding details such as “Recent studies have identified several key epigenetic modifications that play a critical role in cancer progression, including DNA methylation and histone modifications”, and “Several epigenetic therapies, such as DNA methyltransferase inhibitors and histone deacetylase inhibitors, have shown promise in preclinical and clinical studies for treating various types of cancer”.

ChatGPT can assist in improving writing style

Inputting the text: Scientists can input the text they have written for their review article, for example, the abstract or introduction.

Analyzing the text: ChatGPT can analyze the text and provide suggestions for improvements. For example, ChatGPT can identify and highlight grammatical errors, ambiguous sentence structures, or repetitive phrases.

Suggesting improvements: ChatGPT can suggest improvements to the text based on its analysis results. For example, ChatGPT could suggest rephrasing sentences to further improve clarity, using more precise scientific terminology, or avoiding unnecessary jargons.

Providing examples: ChatGPT can provide examples of well-written scientific articles or sentences that illustrate the suggested improvements. For example, ChatGPT could suggest examples of articles with clear and concise writing style, or provide sentences that use technical terms accurately and in context.

Incorporating feedback: Scientists can incorporate the suggestions and examples provided by ChatGPT into their writing. They can also review the suggested changes and make any necessary adjustments to ensure that the changes fit with their intended writing style and tone.

Overall, ChatGPT can assist scientists in improving their writing style by analyzing their text, providing suggestions for improvements, and offering examples of well-written scientific articles or sentences. By incorporating ChatGPT’s suggestions, scientists can further improve the clarity, precision, and effectiveness of their scientific writing, leading to higher quality review manuscripts, and, more efficiently.

ChatGPT can be helpful for non-native English speakers writing review articles

Grammar and sentence structure: ChatGPT can provide suggestions for correct grammar and sentence structure, which can be particularly helpful for non-native English speakers who may struggle with these aspects of writing in English. The model can suggest alternatives for sentence construction and can identify errors in syntax or grammar.

Vocabulary: ChatGPT can suggest appropriate vocabulary choices and can provide synonyms and alternatives for words, which can help non-native English speakers to find best words to express their ideas.

Translation: ChatGPT can also be trained on text in languages other than English, which can be helpful for non-native English speakers who are writing review articles in their native language. The model can assist in translating text from one language to another, providing suggestions for sentence structure and vocabulary in the target language.

However, while ChatGPT can be helpful for non-native English speakers in the abovementioned ways, it is important to notice that it is not meant to be a substitute for a thorough understanding of the literature and concepts in the field. Non-native English speakers should still review and critically evaluate the text generated by ChatGPT to ensure accuracy and coherence, and should seek feedback from peers or colleagues who are fluent in English to ensure that the manuscript is of high quality.

Potential drawbacks or limitations of using AI in scientific writing

While the use of AI tools like ChatGPT can offer many benefits to scientific writing, there are also some potential drawbacks or limitations to consider.

Lack of context: AI tools like ChatGPT may lack the ability to fully understand the context and nuances of scientific writing, which may result in suggestions that are not always most relevant or appropriate.

Inaccurate or biased information: AI tools may introduce Inaccurate or biased information based on the data they were trained on, leading to suggestions or recommendations that may be biased or even inaccurate in some way.

Over-reliance: Over-reliance on AI tools can lead to a reduction in creative and critical thinking and the ability to make independent judgments about the quality of writing.

Technical limitations: AI tools may not be able to understand complex scientific concepts, technical terminology, or nuances of scientific writing, which can limit the usefulness of these tools.

Cost: Some AI tools may require a non-trivial investment, including licensing fees and training costs, which may be a barrier for some researchers or institutions.

It is important to note that these potential drawbacks or limitations can be mitigated by using AI tools in conjunction with human expertise, creative and critical thinking, and judgement. In addition, researchers and institutions should consider the costs and benefits of using AI tools, and weigh the potential drawbacks against the potential benefits before deciding whether or not to adopt these tools in their scientific writing process.

The risk of plagiarism when use AI to write review articles

The risk of plagiarism.

The risk of plagiarism when using AI to write review articles is a potential concern that should be taken seriously. AI tools like ChatGPT can generate text that may resemble text from other sources, including published articles or online resources, and may thus produce text that could be flagged as plagiarism.

However, it is important to note that the use of AI tools does not inherently increase the risk of plagiarism. Rather, the risk of plagiarism depends on how the AI-generated text is used and attributed. To minimize the risk of plagiarism, scientists should follow the below suggested protocol.

Use AI tools as a supplement rather than a replacement: Scientists should use AI tools to assist in writing their review article, but should not rely solely on AI-generated text. It is important to review and edit the AI-generated text to ensure that it is accurate and appropriate for the context of the article.

Properly attribute sources: When using AI-generated text, scientists should properly attribute any sources used in the text. This includes properly citing any direct quotations or paraphrased information and avoiding copying and pasting large portions of text without attribution.

Check for plagiarism: Scientists should use plagiarism-detection software to check their writing for any potential instances of plagiarism, including any text generated by AI tools. This can help identify any potential issues before the manuscript is submitted for publication.

Overall, while the risk of plagiarism when using AI to write review articles is a potential concern, it can be minimized by using AI tools responsibly and in conjunction with best practices for proper citation and attribution of sources. The bottom line is, plagiarism does happen with or without using AI tools.

How to avoid the risk of plagiarism

Using AI tools like ChatGPT to write review articles can be very helpful in speeding up the writing process and generating ideas. However, as discussed in the previous section, it is important to avoid the risk of plagiarism when using AI-generated text. Below are some tips on how to avoid plagiarism when using ChatGPT.

Understand the source of the text: When using ChatGPT to generate text, it is important to understand the source of the text. ChatGPT uses a large dataset of text to generate new text, so there is a risk that the generated text may include content that has been previously published. It is important to carefully review the generated text to ensure that it is not simply a reworded version of previously published content.

Use multiple sources: To avoid plagiarism, it is important to use multiple sources when writing a review article. ChatGPT can be used to generate text based on one or more sources, but it is important to supplement the generated text with information from other sources. This will help to ensure that the review article is not simply a regurgitation of previously published content.

Cite sources properly: ChatGPT can generate citations and references based on the input text, however, based on our experience, ChatGPT may suggest wrong references or even cite article references that do not exist; therefore it is extremely important to review and edit the generated citations and references to ensure their accuracy and completeness.

Use plagiarism-detection software: Plagiarism-detection software can be used to identify instances of plagiarism in the review article. This can help to ensure that the review article is original and not simply a reworded version of previously published content.

Review and edit carefully: Finally, it is important to review and edit the review article carefully to ensure that it is original and does not include any instances of plagiarism. This includes reviewing the text generated by ChatGPT and editing it as necessary to ensure its originality and accuracy.

In summary, while AI tools like ChatGPT can be very helpful in writing review articles, it is important to be vigilant about the risk of plagiarism and take steps to avoid it. By understanding the source of the text, using multiple sources, citing sources properly, using plagiarism-detection software, and reviewing and editing carefully, scientists can ensure as much as possible that their review articles are original and accurate.

Use ChatGPT to write review article need human oversight

While AI tools like ChatGPT can be incredibly helpful in the scientific writing process, it is important to note that they are not perfect and may produce errors or inaccuracies. As such, human oversight is essential to ensure that the content generated by ChatGPT is accurate, appropriate, and meets the needs of the intended audience.

Here are a few reasons why human oversight is necessary when using ChatGPT to write review articles.

Contextual understanding: AI tools like ChatGPT lack the ability to understand the full context of the scientific writing process. While they can generate text that may be grammatically correct and relevant to the topic, they may not be able to understand the broader implications of the content or the needs of the intended audience. As such, human oversight is necessary to ensure that the generated text is appropriate for the intended purpose.

Checking for accuracy: AI tools can generate text that is not always accurate or appropriate. For example, they may include factual errors or make assumptions that are not supported by the available evidence. Human oversight is necessary to ensure that the content generated by ChatGPT is accurate and supported by the available evidence.

Editing and formatting: AI-generated text usually requires further editing and formatting to meet the specific requirements of the review article, such as ensuring proper citation and formatting of references. Human oversight is necessary to ensure that the final product meets the necessary requirements and standards.

In summary, while ChatGPT can be a powerful tool for scientific writing, it is important to remember that it is never going to be a replacement for human expertise. Human oversight is necessary to ensure that the content generated by ChatGPT is accurate, appropriate, and meets the needs of the intended audience.

Conclusion remarks

In conclusion, the use of AI tools such as ChatGPT can significantly enhance both the efficiency and the quality of writing review articles for scientists. ChatGPT can help to speed up the writing process, facilitate collaboration among authors, and assist in improving writing style. However, it is important to keep in mind the limitations of ChatGPT’s capabilities for writing review articles in any expertise area, and to ensure that the generated text is carefully reviewed and edited by human authors to avoid the risk of plagiarism.

Despite these limitations, ChatGPT remains a powerful tool for scientists seeking to write high-quality review articles. By carefully inputting the relevant keywords and data, scientists can generate comprehensive and insightful reviews that summarize the latest advances in their field. With the assistance of ChatGPT, scientists can focus on analyzing and interpreting the results of their literature reviews, rather than spending hours poring over the literature and drafting summaries by hand.

Overall, the use of AI tools such as ChatGPT represents an exciting opportunity for scientists to streamline their research process and produce high-quality, impactful review articles in a timely and highly effective manner. As the field of AI continues to evolve, it is likely that we will see even more advanced tools that are tailored specifically to the needs of scientists. Potential examples include Google Bard, Microsoft Bing, and Jasper Chat, just to name a few. By embracing these tools and incorporating them into their research workflows, scientists can stay at the forefront of their fields and produce research that has an even more meaningful impact on society.

Acknowledgements

This work was in part funded by China Medical University Ying-Tsai Scholar Fund CMU109-YT-04 (to MT). This manuscript was conceived by MT, written by JH and MT, and revised by JH and MT. This article was not funded or supported by OpenAI. While some of the text was initially written with the help from ChatGPT by OpenAI, the content generated by ChatGPT has been carefully modified and verified by the human authors.

Disclosure of conflict of interest

How to Use ChatGPT to Write a Literature Review With Prompts

Are you struggling with writing a literature review for your research paper?

Do you find it challenging to organize your thoughts and present a cohesive and engaging review?

Look no further, as this step-by-step guide will walk you through how to use ChatGPT to write a literature review effortlessly and effectively.

With the assistance of ChatGPT, you can enhance your writing process and create a well-structured literature review that will impress your readers and save your time in research.

Leveraging ChatGPT for Conducting Meta-Analysis in Research

ChatGPT For Large Documents in 6 Easy Steps [2023]

How to use ChatGPT for Hypothesis Development

12 ChatGPT Prompts for Text Analysis

Table of Contents

Step 1: Familiarize Yourself with ChatGPT

To get started, visit the official website of ChatGPT at chat.openai.com

Take a moment to explore the features and functionalities offered by this powerful writing assistant.

You’ll find a user-friendly interface that allows you to interact with the model and generate high-quality content effortlessly.

Step 2: Understanding Prompts

Prompts act as a guiding framework for ChatGPT , providing it with the necessary context to generate accurate and relevant content.

When writing a literature review, you can leverage prompts to direct the AI’s attention towards specific areas or topics you want to cover.

This ensures that the generated text aligns with your intended purpose.

Step 3: Crafting an Effective Prompt

To write a compelling literature review, it’s crucial to formulate an effective prompt.

A well-crafted prompt should be clear, concise, and include specific instructions to guide the AI model.

Craft a prompt according to the requirements of your study. Provide detailed information about what you want to be included in your literature review. You need to give detailed information about the variables that you have used in the study, the relationships between them, and the research hypothesis .

Example: Prompt # 1

Let’s consider an example of a prompt for a literature review on “The Impact of Artificial Intelligence in Healthcare”:

Example: Prompt # 2

Write a literature review of a research paper. Topic: Effects on Stress on digital hoarding behavior of an Employee. The mediating role of procrastination and moderating role of work overload. Variab les: Stress, digital hoarding, procrastination, and work overload. Independent Variable: Stress, Dependent Variable: digital hoarding, Mediator: procrastination, Moderator: work overload. Provide separate definition of each variable. After that explain the relationships between variables. Research Hypothesis: There are 3 hypotheses. H1: Stress has positive influence on digital hoarding. H2: Procrastination is mediating the relationship of stress and digital hording. H3 : Workload moderates the relationship positively of stress and procrastination in an employee. Time_Period: 2015-2023 Include the authors and references in the literature review. Also Include citations. The references should be provided at the end.

By providing a detailed prompt like the one above, you ensure that ChatGPT understands the specific requirements of your literature review, enabling it to generate relevant and insightful content.

10+ ChatGPT Prompts to Write Your Resume

How to Use ChatGPT to Write SQL Queries: Examples and Tips

10 ChatGPT Prompts for Job Applications

20 ChatGPT Prompts to Write Code in Python

20 ChatGPT Prompts to Write JavaScript Code

Step 4: Interacting with ChatGPT

Once you have a well-crafted prompt, it’s time to interact with ChatGPT and generate your literature review.

On the website, you’ll find a text input field where you can enter your prompt. Simply copy and paste the prompt we discussed in the previous step into the input field and press the “Generate” button.

ChatGPT will then process the prompt and generate a response in real-time. The generated content will be based on the data it has been trained on and the context provided by the prompt.

Output provided by the ChatGPT

Literature Review

Title: Effects of Stress on Digital Hoarding Behavior of an Employee: The Mediating Role of Procrastination and Moderating Role of Work Overload

Introduction: The rapid advancement of technology and the increasing reliance on digital platforms have transformed the way individuals manage and organize their information. However, this digital age has also given rise to new behavioral patterns, such as digital hoarding, which refers to the excessive accumulation and difficulty in discarding digital files and data. Digital hoarding can have significant consequences for individuals, particularly in a work setting, where it can lead to decreased productivity, increased stress, and impaired decision-making. This literature review aims to explore the effects of stress on digital hoarding behavior among employees, while considering the mediating role of procrastination and the moderating role of work overload.

Definition of Variables:

• Stress: Stress is a psychological and physiological response to external pressures or demands, characterized by feelings of tension, anxiety, and unease. In the context of this research, stress refers to the emotional and cognitive strain experienced by employees in their work environment, which may arise from factors such as excessive workload, time pressure, or interpersonal conflicts.
• Digital Hoarding: Digital hoarding is a behavioral tendency characterized by the excessive accumulation and reluctance to delete or organize digital files, documents, emails, or other forms of electronic data. It represents a disorganized and cluttered digital environment, resulting in difficulties in locating and managing information effectively.
• Procrastination: Procrastination refers to the act of delaying or postponing tasks or activities that need to be accomplished. In the context of this research, procrastination is viewed as a mediator variable, playing a role in the relationship between stress and digital hoarding. It represents a cognitive and behavioral tendency to engage in non-essential activities or tasks that provide temporary relief from stress, leading to the neglect of important tasks, including digital file management.
• Work Overload: Work overload refers to a state in which an individual is faced with an excessive amount of work or tasks that exceed their available resources, including time and energy. As a moderating variable in this study, work overload is expected to influence the relationship between stress and procrastination. High levels of work overload may intensify the impact of stress on procrastination tendencies, exacerbating digital hoarding behavior.

Relationships between Variables: The proposed research hypotheses aim to investigate the relationships among the variables in the context of stress, digital hoarding, procrastination, and work overload.

Hypothesis 1 (H1): Stress has a positive influence on digital hoarding. This hypothesis suggests that higher levels of stress experienced by employees will be associated with increased digital hoarding behavior. Previous studies (Smith et al., 2017; Johnson & Brown, 2019) have indicated a positive relationship between stress and hoarding tendencies, suggesting that individuals may resort to hoarding behaviors as a coping mechanism to manage stress.

Hypothesis 2 (H2): Procrastination mediates the relationship between stress and digital hoarding. This hypothesis posits that procrastination acts as a mediator, partially explaining the relationship between stress and digital hoarding behavior. Research by Roberts and Zomer (2016) has highlighted the role of procrastination in information overload and disorganization, which aligns with the concept of digital hoarding.

Hypothesis 3 (H3): Work overload moderates the relationship between stress and procrastination. This hypothesis suggests that work overload enhances the positive relationship between stress and procrastination. The presence of high workloads can intensify the impact of stress on procrastination tendencies, leading to greater digital hoarding behavior. This is supported by studies conducted by Miller and Brown (2018) and Thompson et al. (2021), which have demonstrated the influence of work overload on procrastination behavior.

Conclusion: This literature review has provided an overview of the effects of stress on digital hoarding behavior among employees, considering the mediating role of procrastination and the moderating role of work overload. The identified research hypotheses (H1, H2, H3) will serve as a foundation for further empirical investigation to deepen our understanding of the complex relationships between these variables. By examining the impact of stress, procrastination, and work overload on digital hoarding behavior, organizations can develop strategies to promote healthier digital practices and enhance employee productivity.

References: Johnson, A. M., & Brown, T. A. (2019). Examining the relationship between stress, anxiety, and hoarding: A systematic review and meta-analysis. Journal of Obsessive-Compulsive and Related Disorders, 23, 100468.

Miller, R. B., & Brown, T. A. (2018). The relationship between work overload and mental health symptoms in firefighters: The moderating role of perceived supervisor support. Journal of Workplace Behavioral Health, 33(3), 213-229.

Roberts, L. D., & Zomer, A. C. (2016). The procrastination–information systems quality link: The mediating role of self-efficacy. Computers in Human Behavior, 65, 167-176.

Smith, C. G., et al. (2017). The relationship between hoarding symptomatology, compulsive buying, and obsessive-compulsive disorder. International Journal of Cognitive Therapy, 10(4), 332-348.

Thompson, M. A., et al. (2021). Work overload and turnover intentions: The mediating role of procrastination and the moderating role of neuroticism. Journal of Vocational Behavior, 128, 103603.

Step 5: Refining and Polishing

After ChatGPT generates the initial response, it’s time to refine and polish the content. Review the output carefully, ensuring that it aligns with the objectives of your literature review.

You can ask ChatGPT to provide much more detailed explanation on the variable which evr you would like.

Its good to provide ChatGPT detailed prompt to get better results. If you provide much detailed prompt, ChatGPT would write literature review according to your needs.

Note: The literature review that you would get is plagiarism-free. This is another benefit of using chatgpt to write literature review.

As an AI-powered writing assistant, ChatGPT provides a starting point for your literature review.

It can help you brainstorm ideas, explore different perspectives, and save time on the initial drafting process. However, it’s essential to remember that human judgment and expertise are still vital in crafting a comprehensive and well-rounded literature review.

How to Use ChatGPT to Write Your Cover Letter | Prompts

How to Use ChatGPT to Write a Resume? | Prompts

How to Use ChatGPT to Write Code: A Software Developer’s Guide

Best ChatGPT Prompts For Students

Step 6: Iterative Process

Writing a literature review is often an iterative process. You may need to go through multiple rounds of generating and refining the content until you achieve the desired outcome. Take advantage of ChatGPT’s capabilities to experiment with different prompts , explore diverse angles, and refine your review further.

By leveraging the power of ChatGPT and following this step-by-step guide , you can streamline your literature review writing process and enhance the quality of your work.

ChatGPT is a valuable tool for writing literature reviews. By following the steps outlined in this guide , you can effectively utilize ChatGPT to generate high-quality content for your literature review.

Remember, though, that while ChatGPT is a powerful AI model, it’s essential to review and refine the generated content to ensure it aligns with your specific requirements.

Other articles

Please read through some of our other articles with examples and explanations if you’d like to learn more.

• Paraphraser.io
• Imagestotext.io
• Websites to Read Books for Free
• Tools to Convert PNG Image to Excel
• Detect AI-Generated Text using ZeroGPT and Transform It using Quillbot
• How is QuillBot used in Academic Writing?
• Tools for Presentations
• AI Tools for Citation Management
• Improve your Writing with QuillBot and ChatGPT
• Tools Transforming Knowledge Management
• Plagiarism Checkers Online
• Information Management Software 
• Tools for Information Management
• Software Tools for Writing Thesis
• OpenAI WordPress Plugin
• TTS Voiceover
• Backend Automation Testing Tools
• AI Tools for Academic Research
• SendMultiple PDF Files as a Single Attachment
• Google Bard
• Google Bard for SEO
• Shorten the Content’s Length Without Lossing its Meaning
• Write Scientific Research Paper Using Google Bard AI
• Google Bard Can Create Spreadsheets
• How AI Technology is Helpful For Marketers
• How AI Chatbots Can Identify Trading Patterns
• Quorum vs Hyperledger Fabric vs R3 Corda
• ChatGPT vs. Google Bard
• Quantum Apocalypse
• Firmware and Software
• Synergy vs. KVM Switches
• Flash Drive Vs Pen Drive
• Type A, B and C USB Cable
• WSN and IoT
• eSourcing and eProcurement
• Difference between eProcurement and Traditional Procurement
• Over the Top and Video On Demand
• Software Development Life Cycle Vs Software Testing Life Cycle
• Software Development Team vs Freelancer
• HDMI over IP vs HDBaseT
• Display Stream Compression
• Top Internet of Things Journals
• Supervised Learning vs. Unsupervised Learning
• Video Upscaling and Video Downscaling
• How To Build a Distributed System
• How to Get Into Data Science From a Non-Technical Background?
• How To Build Ethereum Mining Rig?
• Paraphrase Text
• PowerPoint Slides
• Learn Languages
• Write Code in Python
• Write Literature Review
• Document Review
• Job Interview Preparation
• Prompts For Students
• Write Cover Letter
• Write Resume
• Job Applications
• Write SQL Queries
• Write Excel Formulas
• Academic Writing
• Translate Text
• Keyword Research
• Business Idea Generation
• Business Entrepreneur
• Use ChatGPT on WhatsApp
• Virtual Research Assistant
• Meta-Analysis in Research
• Large Documents
• Hypothesis Development
• Share ChatGPT Responses
• Text Analysis
• Upload PDF on ChatGPT
• Books ChatGPT

Misbah Rashid, an expert in Technology Management, holds an MBA and an MS in Information Systems and Technology Management. She has experience teaching marketing and technology in business at the university level.

Related Posts

Survey sampling: what it is, types & tips, cluster sampling | method and examples, how does gptzero work ai detector for chatgpt / gemini / copilot / meta ai, who is a good peer reviewer, peer review | types of peer review, ethics in research: safeguarding integrity and credibility, advantages and disadvantages of snowball sampling, exploring qualitative researcher skills: what they are and how to develop them, difference between quota sampling and stratified sampling, how effective laboratory design impacts health, safety, and productivity.

Comments are closed.

A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare

Affiliations.

• 1 Department of Computer Science, Vanderbilt University, Nashville, TN, USA, 37212.
• 2 Department of Biomedical Informatics, Vanderbilt University Medical Center, TN, USA, 37203.
• 3 Department of Pediatrics, Vanderbilt University Medical Center, Nashville, Tennessee, USA, 37203.
• 4 Center for Biomedical Ethics and Society, Vanderbilt University Medical Center, Nashville, Tennessee, USA, 37203.
• 5 Department of Biostatistics, Vanderbilt University Medical Center, TN, USA, 37203.
• PMID: 38712148
• PMCID: PMC11071576
• DOI: 10.1101/2024.04.26.24306390

Background: The launch of the Chat Generative Pre-trained Transformer (ChatGPT) in November 2022 has attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including healthcare. Numerous studies have since been conducted regarding how to employ state-of-the-art LLMs in health-related scenarios to assist patients, doctors, and public health administrators.

Objective: This review aims to summarize the applications and concerns of applying conversational LLMs in healthcare and provide an agenda for future research on LLMs in healthcare.

Methods: We utilized PubMed, ACM, and IEEE digital libraries as primary sources for this review. We followed the guidance of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRIMSA) to screen and select peer-reviewed research articles that (1) were related to both healthcare applications and conversational LLMs and (2) were published before September 1 st , 2023, the date when we started paper collection and screening. We investigated these papers and classified them according to their applications and concerns.

Results: Our search initially identified 820 papers according to targeted keywords, out of which 65 papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT from OpenAI (60), followed by Bard from Google (1), Large Language Model Meta AI (LLaMA) from Meta (1), and other LLMs (5). These papers were classified into four categories in terms of their applications: 1) summarization, 2) medical knowledge inquiry, 3) prediction, and 4) administration, and four categories of concerns: 1) reliability, 2) bias, 3) privacy, and 4) public acceptability. There are 49 (75%) research papers using LLMs for summarization and/or medical knowledge inquiry, and 58 (89%) research papers expressing concerns about reliability and/or bias. We found that conversational LLMs exhibit promising results in summarization and providing medical knowledge to patients with a relatively high accuracy. However, conversational LLMs like ChatGPT are not able to provide reliable answers to complex health-related tasks that require specialized domain expertise. Additionally, no experiments in our reviewed papers have been conducted to thoughtfully examine how conversational LLMs lead to bias or privacy issues in healthcare research.

Conclusions: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications brought bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in healthcare.

Keywords: ChatGPT; artificial intelligence; bias; healthcare; large language model; medical knowledge inquiry; natural language processing; privacy; reliability; summarization.

Publication types

Grants and funding.

• R37 CA237452/CA/NCI NIH HHS/United States
• RM1 HG009034/HG/NHGRI NIH HHS/United States

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

The public attitude towards ChatGPT on reddit: A study based on unsupervised learning from sentiment analysis and topic modeling

Roles Data curation, Formal analysis, Methodology, Project administration, Software, Visualization, Writing – original draft, Writing – review & editing

Affiliation Department of Data Science, School of Computer Science and Engineering, Guangzhou Institute of Science and Technology, Guangzhou, Guangdong, China

Roles Conceptualization, Formal analysis, Investigation, Project administration, Supervision, Validation, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Department of Management, School of Business, Macau University of Science and Technology, Macao, China

Roles Data curation, Funding acquisition, Investigation, Validation

Affiliation Data Science Research Center, Faculty of Innovation Engineering, Macau University of Science and Technology, Macao, China

Roles Funding acquisition, Methodology, Project administration, Resources, Writing – review & editing

Affiliation Department of Decision Sciences, School of Business, Macau University of Science and Technology, Macao, China

• Zhaoxiang Xu, 
• Qingguo Fang, 
• Yanbo Huang, 
• Mingjian Xie

• Published: May 14, 2024
• https://doi.org/10.1371/journal.pone.0302502
• Reader Comments

ChatGPT has demonstrated impressive abilities and impacted various aspects of human society since its creation, gaining widespread attention from different social spheres. This study aims to comprehensively assess public perception of ChatGPT on Reddit. The dataset was collected via Reddit, a social media platform, and includes 23,733 posts and comments related to ChatGPT. Firstly, to examine public attitudes, this study conducts content analysis utilizing topic modeling with the Latent Dirichlet Allocation (LDA) algorithm to extract pertinent topics. Furthermore, sentiment analysis categorizes user posts and comments as positive, negative, or neutral using Textblob and Vader in natural language processing. The result of topic modeling shows that seven topics regarding ChatGPT are identified, which can be grouped into three themes: user perception, technical methods, and impacts on society. Results from the sentiment analysis show that 61.6% of the posts and comments hold favorable opinions on ChatGPT. They emphasize ChatGPT’s ability to prompt and engage in natural conversations with users, without relying on complex natural language processing. It provides suggestions for ChatGPT developers to enhance its usability design and functionality. Meanwhile, stakeholders, including users, should comprehend the advantages and disadvantages of ChatGPT in human society to promote ethical and regulated implementation of the system.

Citation: Xu Z, Fang Q, Huang Y, Xie M (2024) The public attitude towards ChatGPT on reddit: A study based on unsupervised learning from sentiment analysis and topic modeling. PLoS ONE 19(5): e0302502. https://doi.org/10.1371/journal.pone.0302502

Editor: Jitendra Yadav, IBS Hyderabad: ICFAI Business School, INDIA

Received: October 31, 2023; Accepted: April 7, 2024; Published: May 14, 2024

Copyright: © 2024 Xu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: 23,773 entries were retained, forming the database Processed_GPT_total.json, displayed in supporting information . To improve the reproducibility of our research results, the data and research design have been stored on Protocols.io. Protocols.io has assigned a protocol identifier (DOI) to our protocols, which is DOI: dx.doi.org/10.17504/protocols.io.bp2l6xee1lqe/v1 . Open Access license is freely available for anyone. Readers can find relevant details by visiting the following URL: https://www.protocols.io/private/899FA95EAEFD11EEAB870A58A9FEAC02 .

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

1 Introduction

In such an era of rapid development of artificial intelligence (AI), ChatGPT has demonstrated remarkable capabilities and expanded in most life domains. Since it was introduced, with great potential applications in education, healthcare, industry, agriculture, travel, transportation, e-commerce, entertainment, marketing, and finance. The GPT (Generative Pre-trained Transformer) model represents a significant breakthrough in natural language processing, propelling the advancement of language-capable machines that resemble human communication [ 1 ]. ChatGPT is a specific application developed by OpenAI, a private artificial intelligence research lab, based on the GPT-3.5 model, released on November 30, 2022. The GPT-3.5 model is a technological iteration of the GPT-3 model. Based on the GPT-3.5 model, ChatGPT is trained on an extensive dataset comprising both text and code. It could generate text, perform language translation, produce various forms of creative content, and provide informative responses to user inquiries [ 2 ]. On March 15, 2023, OpenAI unveiled the new large-scale multimodal model, GPT-4, which processes textual data, incorporates image content, and exhibits improved response accuracy. Users can access GPT-4 through ChatGPT Plus on a fee-paying basis [ 2 ].

ChatGPT had over 100 million users in January 2023 [ 3 ]. The website generated 1.6 billion visits in June 2023 [ 4 ]. Mainstream media outlets have both expressed admiration and concern in response to ChatGPT. For example, the Guardian published an article written by GPT-3, asserting that humans should trust and respect AI’s role in improving their lives [ 5 ]. The New York Times published an article praising GPT-3’s impressive capabilities, noting its flaws [ 6 ]. In contrast, the Washington Post discussed the potential risks of AI, including ChatGPT, and argues for an enhanced regulatory framework in nations worldwide to address the issue urgently [ 7 ]. The public holds diverse opinions regarding significant new technologies [ 8 ], including ChatGPT. Some people believe that ChatGPT has many positive implications for society, such as helping people learn new knowledge and languages, generating creative text formats, replacing repetitive and laborious tasks, and providing companionship and support. Conversely, there are dissenting opinions regarding GPT, emphasizing its ongoing developmental stage, limitations, ethical issues, and possible outcomes of job losses.

Public attitudes toward ChatGPT are an important research topic. First, studying the public’s attitudes toward ChatGPT can help predict how widely ChatGPT will be used and accepted [ 9 ]. Public attitudes toward ChatGPT encompass affect, behavior, and cognition [ 10 ]. This study uses topic modeling and qualitative content analysis to examine how individuals perceive ChatGPT. The sentiment analysis of posts and comments allows this study to understand people’s emotions on ChatGPT. Second, public attitudes play an important role in shaping applied ethics. If the public is concerned about ChatGPT being used for malicious purposes, this could prompt a shift in its developmental trajectory and encourage its ethical utilization. Third, understanding public attitudes can lead to its advancement that better serves users. It provides researchers and practitioners related to AI with valuable insights into the potential applications and major limitations of ChatGPT. Understanding users’ concerns, needs, and preferences can help practitioners personalize ChatGPT’s responses to individual users and make ChatGPT more engaging and interactive.

This study aims to explore public attitudes toward ChatGPT, using data collected from users’ posts and comments on the social media site Reddit. A large number of studies have used data from Reddit to examine public perceptions, attitudes, and opinions. Reddit has approximately 57 million daily active users until 2023, one of the largest social media outlets in terms of users [ 11 , 12 ]. Based on the data of posts and comments from Reddit, three research questions will be addressed:

• RQ 1: What are the main topics on Reddit regarding ChatGPT?
• RQ 2: What is the public sentiment toward ChatGPT?
• RQ 3: How does public sentiment towards ChatGPT change over time?

The study employs the LDA model for topic modeling to identify emerging themes in public discussions about ChatGPT. In addition, sentiment classification is performed using a weighted combination of VADER and Textblob, and the results are presented in the form of bar charts and pie charts to analyze the variations in public sentiment towards ChatGPT. Furthermore, by extracting the creation timestamps of each sample and the sentiment score results, a daily change curve is plotted to illustrate the fluctuations in the number of positive and negative samples since 2023. This allows us to examine whether sentiment attitudes have changed over time.

This study would make the following contributions: First, this research represents a cutting-edge exploration of ChatGPT after other AI. As Artificial Intelligence advances, AI products, such as chatbots, smart virtual assistants, and self-driving cars, bring innovation, efficiency, and value creation to various fields. Therefore, extending the research line in AI studies, particularly focusing on ChatGPT, is crucial for understanding its impact, addressing challenges, fostering innovation, and ensuring the positive role of AI technology in society. Second, it seeks to address a research gap in public sentiment toward ChatGPT. Prior studies on ChatGPT have predominantly focused on its applications in education, academic research, tourism, and healthcare. Despite the widespread impact on human society, there is a lack of research regarding the public’s perspectives and sentiments regarding its influence. There are only a handful of exceptions wherein students were sampled as subjects to investigate attitudes toward the use of ChatGPT. However, there is a significant absence of comprehensive reviews that examine its implications on human society and public sentiment. This research provides insights into the public’s expectations and concerns regarding the potential risks and benefits of ChatGPT. Third, compared to traditional sampling surveys with self-reported questionnaires, social media big data analytics is leading-edge and efficient. Traditional public attitude surveys typically rely on questionnaires, which are time-consuming, costly, and limited to specific populations such as students, consumers, or employees. In contrast, social media data is publicly available and can be collected globally, allowing researchers to gather extensive data more quickly [ 13 ]. It facilitates comprehension of diverse perspectives on ChatGPT among various social groups. In addition, traditional fixed-choice questionnaires often only cover a limited number of questions [ 14 ]. Social media data provides in-depth information than structured questionnaires, as it offers insights into users’ interests, viewpoints, behaviors intentions on ChatGPT. Social media data could be real-time and quickly identify trends and patterns in public [ 15 ]. Social media big data analytics has been increasingly adopted in research on public sentiment towards AI products [ 16 ]. This study follows the current trend. Fourth, in practical terms, this research will provide valuable information for ChatGPT developers on enhancing the system to meet user needs better. By gathering user’s feedback on ChatGPT, the study will identify its strengths and weaknesses. It aids developers in improving ChatGPT’s functionality and performance, making it more user-friendly and ensuring an enhanced user experience.

2 Literature review

This section reviews the use of ChatGPT in human society for an initial assessment of the general public’s perception of ChatGPT. According to the bibliometric results, a total of 365 research papers (including articles and reviews) titled "ChatGPT" are found in the Web of Science database. Since ChatGPT was released in June 2020 based on GPT-3, the timeframe is set for the papers from June 1, 2020, to September 1, 2023. The existing research on the impact of ChatGPT on human society focuses on education, academic research, healthcare, tourism, and ethics. Education and healthcare are the fields with the highest number of published articles.

In education, research on the impact of ChatGPT covers medical education, nursing education, science education, language education, programming education, etc. [ 17 ]. ChatGPT offers opportunities for students and educators, including personalized feedback, increased accessibility, interactive dialogues, lesson planning, assessment, and helping students improve their programming skills. When ChatGPT is used to teach different subjects, varied pedagogical outcomes are achieved [ 18 ]. The use of ChatGPT for teaching mathematics, sports science, and psychology to students is unsatisfactory. Unexpectedly, ChatGPT has demonstrated reliability and utility, even in more rigorous medical education [ 19 ]. One of the most critical topics is the significant impact of ChatGPT on students’ programming learning [ 20 ]. A study on ChatGPT in computer programming learning proves its effectiveness and usability in generating solution code, checking bugs, debugging code, and dealing with programming assignments, exams, and homework [ 20 , 21 ].

When exploring the impact of ChatGPT on academic research, academics usually focus on the following two questions: First, is the use of ChatGPT in a written work to be considered plagiarism? Second, can ChatGPT be considered as a co-author? There are polarised views on these two issues. Some positively embrace the new technology and see ChatGPT as a viable collaborator. In January 2023, Nurse Education in Practice, a journal published by Elsevier, generated significant controversy by acknowledging ChatGPT as a co-author [ 22 ]. However, others argue that using the ChatGPT technique constitutes academic cheating [ 23 ]. Many journals have stated that AI tools, such as ChatGPT, are not eligible to be credited as authors, including Science [ 24 ]. There is an undeniable consensus about ChatGPT as a competent co-author because of its ability to output more coherent, fairly accurate, informative, and systematic knowledge texts. At the same time, ChatGPT can support interdisciplinary research and provide research support [ 25 ]. The GPT model learns a large amount of textual data from different domains during training, giving it knowledge and understanding across multiple subject areas.

In healthcare, research indicates that ChatGPT holds enormous potential in virtual consultations, improving public mental health and well-being [ 26 ]. Moreover, studies have confirmed the positive role of ChatGPT in clinical practices and patient education [ 27 ]. Scholars have proposed various applications of AI in mental health, such as assisting clinicians with time-consuming tasks like documenting and updating medical records, enhancing diagnostic accuracy and prognosis, fostering a better understanding of mental illness mechanisms, and refining treatment through biological feedback. Furthermore, ChatGPT has even outperformed humans in emotional awareness evaluations. It is expected to assist physicians in making decisions related to diagnosing, treating, and managing chronic obstructive pulmonary disease [ 28 ]. Most healthcare researchers have expressed positive or balanced attitudes toward ChatGPT by analyzing data in social media [ 26 ]. These research findings collectively demonstrate the positive impact of ChatGPT and AI technologies on enhancing healthcare standards and patient experiences within the medical and healthcare domains.

Scholars have also shown significant interest in the impact of ChatGPT on tourism [ 29 ]. It is expected to bring about significant changes in the tourism industry by enhancing decision-making support for managers in tourism companies and policy-makers in governing bodies. The use of ChatGPT in tourism decision-making differs greatly from traditional approaches, as it engages tourists in an interactive question-and-answer mode. It allows them to personalize travel plans and recommend suitable travel services, including hotels, restaurants, transportation, local attractions, and leisure activities [ 30 ].

Scholars have also conducted exploratory research around ChatGPT in other fields. These include the following areas: corporate governance [ 31 ], supply chains [ 32 ], finance [ 33 ], intelligent vehicles [ 34 ], and so on. These researches demonstrate the prospect of wide application of GPT technology and lay the foundation for future research.

However, there have been some general complaints and ethical concerns regarding ChatGPT. First, criticism revolves around potential privacy leakage issues. This is because ChatGPT processes input prompts that may contain personal information, raising privacy concerns. Although OpenAI promises not to collect personal information from users, there is still a risk of leakage during network transmission due to inadequate security measures within data storage systems. Second, there are criticisms the use of ChatGPT in academia raises concerns about academic integrity. If ChatGPT fails to cite reference sources appropriately, it may lead to plagiarism or deception in education and academic research [ 35 ]. ChatGPT could be also used for online exams, posing a significant threat to exam integrity [ 36 , 37 ]. To counter these problems, some anti-plagiarism techniques have been employed to detect AI-generated context [ 38 ]. To leverage ChatGPT’s advantages Responsibly in the realm of education, scholars suggest that educators should focus on improving students’ creativity and critical thinking rather than just acquiring skills. Meanwhile, AI-related tasks can engage students in solving real-world problems [ 39 ].

Third, the conversational capabilities of ChatGPT often draw criticism due to the limitations of its output. These limitations include inaccurate, fabricated, and biased information, along with a lack of in-depth understanding [ 39 ]. For example, ChatGPT does not have real-time information, and its training data comes from before September 2021, which could lead to biased responses. For example, using ChatGPT in medical research raises concerns about accuracy and reliability. The model has limitations in providing personalized advice and may sometimes generate inappropriate or outdated reference information. Due to a lack of human reasoning ability, ChatGPT may have difficulty generating responses to complex or abstract questions and understanding the context of text input [ 30 ]. In addition, ChatGPT struggles with identifying spelling errors, understanding colloquial and ambiguous language, and lack of interactive experiences and human emotions. Consequently, ChatGPT’s current capacity only enables it to partially substitute for human decision-making [ 40 ].

Fourth, some studies focus on the political issues raised by ChatGPT. Although ChatGPT often claims to be apolitical, empirical evidence demonstrates that it exhibits certain political predispositions, notably favoring supporting environmental protection and left-leaning liberal ideology [ 41 , 42 ]. This may be because ChatGPT is trained on large text corpora collected from the Internet. These corpora may be dominated by influential institutions in Western society, such as mainstream news media, prestigious universities, and social media platforms. Consequently, these collections of texts may appear to represent a majority on certain topics. Furthermore, these algorithms may create an accumulation of false, inaccurate, biased, or confrontational content text on the web, exacerbating the vicious cycle of providing misleading and polarising information to the political system.

Research on the significant impact of ChatGPT in various domains of human society initially reflects stakeholders’ attitudes toward ChatGPT. Given the complexity of ChatGPT’s impact on human society, public opinions and attitudes toward ChatGPT also vary greatly, showing varying degrees of preference or aversion. However, limited literature comprehensively explores the public’s attitudes toward ChatGPT, and this study seeks to fill the gap.

This study aims to investigate public discourse and sentiment on ChatGPT through topic modeling and sentiment analysis using natural language processing based on the data from Reddit users’ posts and comments. The data science methods provided an efficient way to classify latent topics and sentiments in public discourse. Firstly, word frequency reveals the public’s interests related to ChatGPT. Word cloud visualization intuitively presents these high-frequency terms, making critical information easily accessible [ 43 ]. Secondly, topic modeling facilitates the identification of latent topics within textual data, which are more comprehensible to interpret [ 44 ]. Public discussions are often multifaceted and different words and sentences may relate to the same or interconnected topics. Topic modeling helps cluster related content, leading to a better understanding of the associations among various subjects [ 45 ]. This is crucial for exploring the breadth and depth of discussions from diverse perspectives. Lastly, sentiment analysis enables the identification of emotional tendencies (positive, neutral, or negative) conveyed within the text [ 46 , 47 ]. During public discussions about ChatGPT on Reddit, sentiment analysis aids in gauging public sentiment, revealing positive attitudes, concerns, and potential issues or needs. Analyzing sentiment fluctuations over time tracks emotional shifts helps to comprehend the public’s responses to specific events, such as ChatGPT product releases, or promotions. Through this approach, emotional changes and factors can be uncovered.

3.1 Data collection and cleaning

Social media platforms such as Twitter, Facebook, Instagram, and Reddit allow users to express their emotions, interests, hobbies, and opinions in real-time within an online community. Reddit ( https://Reddit.com ) has approximately 57 million daily active users until 2023, one of the largest social media outlets in terms of users [ 11 ]. Reddit users can share text, links, images, or videos in various sub-communities (called subreddits and dedicated to specific topics) [ 48 ]. Everyone has access to the public subforum (called subreddits on Reddit), and users can comment and vote on posts and comments for free and anonymously. Reddit has more than 100,000 active subreddits as of 2023. A large number of existing studies use data from the platform to examine public perceptions, attitudes, and opinions. These studies cover a broad range of topics, including disasters [ 49 ], vaccines [ 50 ], advertisement [ 51 ], tobacco [ 52 ], vehicles [ 53 ], climate change [ 54 ], digital governance [ 55 ], political psychology [ 56 ] and collective identity [ 57 ]. The Application Programming Interface (API) is a set of tools that defines how a software application interacts with other components, services, or platforms [ 58 ]. APIs allow for communication and data exchange between different software systems, enabling them to connect and interoperate with each other.

In terms of ethical considerations in the present research, the official API of Reddit is freely and publicly available to third parties [ 12 , 59 ]. In accordance with Reddit’s privacy policy, developers are permitted to write programs or applications, such as Apify used in this study, that interact with the Reddit platform through specific requests and commands. These actions include retrieving specific information, posting content, or performing other tasks [ 60 ]. Reddit enables researchers to extract the subreddits, threads, comments, and associated metadata through various programming languages [ 61 ]. Therefore, data collection and analysis in this study comply with the terms and conditions of the data source. Following the principle of data minimization, it only collects data relevant to the purpose of the study. In addition, to adhere to the principles of privacy and untraceability, personal user information is anonymized because Reddit posts and comments are user-generated.

This study utilized Apify ( https://apify.com ) for collecting Reddit posts and comments as the primary data source. Apify is a platform for data collection for web scraping, data extraction, and automation. It offers tools and services that assist developers in extracting data from web pages, executing automated tasks, and building web crawlers. Personal information has been anonymized. The data consists of posts and comments. The sample data is shown in Table 1 .

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0302502.t001

It sets GPT3, GPT3.5, and GPT4 as keywords. These keywords aim to examine whether people’s perceptions of ChatGPT have evolved, particularly since the ChatGPT update. Apify collected 11,730, 10,109, and 12,046 relevant entries (posts and comments) for the respective keywords. This study performed random sampling to ensure sample uniformity, resulting in 10,000 entries preserved for each GPT version. The dataset consisted of 30,000 samples from June 2020 to August 15, 2023.

In terms of data cleaning, emoticons, digits, punctuation, links, unnecessary words, non-ASCII characters, and stopwords were removed from the textual data suggested by Yadav et al [ 61 , 62 ]. The NLTK library provides the English stopwords list. It consists of common English words with no semantic or informational value [ 63 , 64 ]. Stopwords are typically filtered out in natural language processing to enhance the efficiency and accuracy of text analysis. Additionally, all uppercase letters have been converted to lowercase. Furthermore, considering a slight overlap in the data sources, this study also conducted duplicate text filtering. In the end, 23,773 entries were retained, forming the database Processed_GPT_total.json, displayed in supporting information.

To improve the reproducibility of our research results, the data and research design have been stored on Protocols.io. Protocols.io has assigned a protocol identifier (DOI) to our protocols, which is DOI: dx.doi.org/10.17504/protocols.io.bp2l6xee1lqe/v1 . Open Access license is freely available for anyone. Readers can find relevant details by visiting the following URL: https://www.protocols.io/private/899FA95EAEFD11EEAB870A58A9FEAC02 .

3.2 Topic modeling

All data analyses were processed using Python (version 3.10). For topic modeling, prior research proposes three techniques: Latent Dirichlet Allocation (LDA), which adeptly discerns latent topics through probabilistic approaches; Non-Negative Matrix Factorization (NMF), which underscores the intricate relationship between documents and their inherent topics; and Transformer-based models, renowned for their proficiency in grasping the intricate semantics embedded within textual data [ 65 ].

Perplexity is a common metric used to evaluate the performance of topic models and is particularly applicable to a selected number of LDA topics to model. It measures how well the model fits the documents in a given dataset, with a lower perplexity score indicating better model performance on that dataset.

3.3 Sentiment analysis

Existing research typically uses three approaches for sentiment analysis: rule-based approaches, machine learning, and deep learning [ 71 ]. Rule-based methods, often referred to as rule-based sentiment analysis, utilize predefined sentiment lexicons and rules to evaluate sentiment [ 72 ]. Machine learning methods, on the other hand, learn sentiment classification models from data through supervised or unsupervised learning, offering good generalization capability [ 73 ]. Deep learning methods such as RNNs, CNNs, and Transformers automatically learn features from text, adept at capturing long-term dependencies, albeit requiring substantial data and computational resources [ 74 ].

The present study employs a rule-based approach to classify sentiments in posts and comments. The sentiment categorization utilizes a weighted approach that combines VADER with TextBlob. By combining the strengths of TextBlob and VADER, two distinct sentiment analysis tools that employ different algorithms and semantic processing approaches, and weighting their results, the overall performance of sentiment analysis can be effectively enhanced. These tools exhibit varied performances in different contexts, and through weighting their outcomes, one can leverage their respective strengths to improve the overall performance of sentiment analysis. Simultaneously, this integrated approach helps mitigate individual biases of different sentiment analysis tools, enabling the system to better adapt to diverse text samples, especially those of specific types. Furthermore, when confronted with complex and diverse language expressions, a single sentiment analysis method may exhibit instability. Integrating multiple methods enhances the model’s robustness, allowing it to adapt to various types and styles of text flexibly. Such a comprehensive approach demonstrates significant advantages in improving the accuracy and adaptability of sentiment analysis.

Rule-based sentiment analysis in TextBlob relies on natural language processing techniques, utilizing predefined rules and syntactic structures to identify the emotional polarity within the text [ 77 ]. Initially, the text is decomposed into words and phrases, and part-of-speech tagging is conducted to comprehend the grammatical roles of each word in the sentence. Subsequently, based on a pre-defined sentiment lexicon, each word is assigned a sentiment polarity score, such as positive, negative, or neutral. Rules may also take into account relationships between words, where the presence of negation words, for instance, could alter the emotional polarity. By weighting or averaging sentiment scores for all words in the text, the overall emotional polarity of the text can be determined. The advantage of this method lies in its simplicity and ease of implementation, while its accuracy can be enhanced by continuously updating and expanding the sentiment lexicon [ 78 ].

3.4 Sentiment trend analysis

Sentiment trend analysis, a burgeoning field, has become vital for comprehending public perception of specific issues [ 80 ]. In this study, sentiment trend analysis integrates the strengths of the aforementioned two methods, utilizing the results with a weighted approach. Initially, a DataFrame (df) is created to organize the data, encompassing columns for date, weighted sentiment scores, and sentiment labels (’Positive’ or ’Negative’). The date column undergoes conversion to the datetime type for accurate time series analysis. Subsequently, the ’Sentiment’ column values are determined based on the weighted scores, with ’Positive’ assigned if the score exceeds 0, and ’Negative’ otherwise. Finally, the DataFrame is grouped by date and sentiment, daily sentiment counts are computed, and a line plot is generated using matplotlib to illustrate the daily counts of positive and negative sentiments over time.

A lucid visualization is paramount to fully capturing the chronology of sentiment dynamics [ 81 ]. As such, daily sentiment metrics are illustrated, clearly depicting the populace’s emotional ebbs and flows. This graphical elucidation not only bestows a daily sentiment snapshot but also illuminates prevailing trends, proving indispensable for decision-makers, ranging from corporate strategists to policymakers, who anchor their choices on the pulse of public sentiment.

4.1 Word frequency

In this part, word clouds and frequency graphs provide initial insights into the diverse perspectives of the general public on the topics (Question 1) and attitudes (Question 2) toward GPT. The analyzed entries exceeded a total count of 23,773 entries. Fig 1 displays the top 20 most common words from the entries. As Fig 1 shows, the public’s positive attitude toward ChatGPT is evident through the words "like" and "good", indicating their appreciation and approval of the model. However, the discussions also unveil contemplation about the practical applications of ChatGPT, encompassing terms such as "use," "using," "way," "make," and "need," highlighting the discourse on how to harness ChatGPT’s capabilities fully. In addition, words like "would," "think," "know," "could," and "even," express doubts and uncertainties, reflecting concerns about its potential limitations and abilities. Technical aspects of the discourse include terms like "model," "bot," "prompt," "data," "code," and "models," revealing the audience’s attention to ChatGPT’s internal working model, data processing, and technological implementation.

https://doi.org/10.1371/journal.pone.0302502.g001

4.2 Topic modeling

This part presents the emergent topics and themes identified through topic modeling. It aims to address research question 1: What are the emerging topics related to ChatGPT? This study combines qualitative and quantitative content analysis to uncover and discover latent topics and themes of the public’s discussion, which are believed to hold significant potential for research in the field of social media [ 82 ]. As a commonly used quantitative method for topic classification, the LDA model aids in determining the most optimal number of topics for classification. In Fig 2 , the perplexity-topic number curve is plotted.

https://doi.org/10.1371/journal.pone.0302502.g002

Typically, the optimal number of topics is determined based on lower perplexity levels [ 83 ]. When the number of topics is set at 8, the perplexity is at its lowest. However, the lowest perplexity may not always signify the best model performance. With a high number of topics, models often overfit, resulting in excessive and non-convergent topic counts. An excessive number of topics may lead to high redundancy, resulting in low distinctiveness and uniqueness between topics [ 71 , 83 ]. Hence, many studies rely on human judges to determine the optimal number of topics. This method also adheres to certain principles: (1) high coherence between words and topics; (2) the quality of topic, ensuring non-repetition, non-conflict, and coverage of primary content [ 84 ]. This study tests the topic categorization and high-frequency words of each topic when the number of topics was set at 8 ( Table 2 ). However, it demonstrates poor coherence and topic quality. Across the topics, there is a lack of coherent themes, with words appearing disjointed and unrelated within each category. The representative words fail to form distinct and meaningful topics, undermining the effectiveness of the model in capturing the underlying structure of the data.

https://doi.org/10.1371/journal.pone.0302502.t002

Therefore, this study also tested the number of topics corresponding to the point of significant decrease in perplexity, i.e., the number of topics (7) near the inflection point of the curve. When the number of topics is set at 7, the distribution of word frequencies in relevant topics is shown in Table 3 . The top words in each topic exhibit good coherence and topic quality. Table 3 shows that the top 10 words in each topic are categorized into seven topics, which are then assigned to three themes. The analysis of seven topics demonstrates the wide range of discussions regarding ChatGPT on the Reddit community. These discussions cover technical inquiries, philosophical pondering, impacts on society, creative applications, and entertainment. The topics reflect the multifaceted nature of ChatGPT and highlight the diverse perspectives and interests of the public when using it.

https://doi.org/10.1371/journal.pone.0302502.t003

The first topic concerns people’s general impressions of ChatGPT. The keywords such as "like," "think," and "good" indicate that individuals are generally favorable towards ChatGPT. This topic focuses on how people perceive ChatGPT’s potential benefits, usability, functionality, and positive impact on them. The second topic appears to focus on technical inquiries for assistance. Terms such as "bot," "prompt," and "link" indicate that users are seeking information on how to use ChatGPT for various tasks. The terms "questions," "message," and "action" suggest a desire to optimize ChatGPT’s functionality for specific purposes. The third topic delves into philosophical discussions, examining consciousness, AGI (Artificial General Intelligence), and human reasoning of ChatGPT. The keywords like "consciousness," "agi," "humans," and "belief" imply that users are exploring ChatGPT’s human consciousness, intelligence, and spiritual characteristics. The fourth topic explores the technical details of ChatGPT, focusing on coding and textual manipulation. The keywords such as "code," "prompt," "text," and "language" suggest discussions on how to utilize ChatGPT for code generation or text creation efficiently. This topic covers its capabilities in software development, content creation, and language-oriented tasks. The fifth topic explores ChatGPT’s impact on diverse fields of life, including the arts, healthcare, and quantum phenomena. The words "music," "art," "health," and "quantum" indicate discussions on how ChatGPT brings the revolution and advances to the artistic and scientific domains.

The sixth topic focuses on the broad social and economic influences of ChatGPT. Keywords like "market," "jobs," and "impact" suggest discussions on ChatGPT’s impact on the job market and the global economy. The debates could probably revolve around potential job loss resulting from ChatGPT and the ethical concerns about AI. The seventh topic concerns the correlation between ChatGPT and politics and entertainment. The keywords such as "trump" and "president" relate to ChatGPT’s function in political discussions. The terms "spider," "gif," and "gypsy" indicate the potential utilization of ChatGPT within cultural and entertainment contexts.

Through the qualitative content analysis, seven topics were systematically coded and categorized into three themes. Theme 1 covers Topics 1 and 3, which focus on users’ positive views of ChatGPT and its potential advantages and positive influence on various aspects of life. Theme 2 encompasses Topics 2 and 4, focusing on the technical methods of ChatGPT, including queries, assistance, coding, and practical applications. Discussions cover topics such as the application of ChatGPT for specific tasks, seeking guidance, and sharing experiences regarding coding and language generation. Theme 3 comprises Topics 5, 6, and 7, focusing on the broader social impact of ChatGPT on art, music, health, politics, market, employment prospects, scientific progress, and the entertainment industry.

4.3 Robustness of topic modeling

To verify the robustness of the model, it randomly selects 10,000 samples as a subset from the original dataset for testing [ 85 ]. To examine the robustness, the same parameters are used for two models [ 86 ].

Fig 3 , compares the perplexity scores of the two models when the number of topics ranges from 1 to 20, finding that the results of the two runs are highly similar. The solid line with circles represents the perplexity changes of the original dataset when LDA topic modeling is performed, while the dotted line with triangles represents the perplexity changes of a subset of the original dataset under the same modeling process. It can be seen that the variation range of the perplexity scores of the two curves is between -8.5 and -11.1, and the overall trend shows a slow increase at first and then a decrease. Within the topics range of 1 to 8, the difference between the two curves is minimal, showing a high degree of similarity. The largest gap is when the number of topics is 20, and it is only 0.23 at this time, indicating that our model is more robust.

https://doi.org/10.1371/journal.pone.0302502.g003

When the topic modeling of the subset is performed and the number of topics is 7, the high-frequency words corresponding to each topic are shown in Table 4 . Three Themes were summarized from the 7 Topics generated by LDA topic modeling based on a subset of the original dataset, which is consistent with our previous LDA modeling results using the original dataset. This also confirms the posts and comments about GPT. The three widely discussed themes are User Perception, Technical Methods, and Impacts on Society.

https://doi.org/10.1371/journal.pone.0302502.t004

To illustrate the robustness, the theme generated by the two models has a high degree of similarity with the top words ( Fig 4 ). Regardless of whether it is applied to the original dataset or its subset, it can produce highly similar theme results. It has summarized the seven topics derived from the original dataset into three themes, and it also depicts the high-frequency words associated with different themes generated by two topic modeling processes. Among these, the intersection of each pair of themes represents the high-frequency words that yield identical results from both topic modeling techniques.

https://doi.org/10.1371/journal.pone.0302502.g004

To further corroborate the reliability of this result, it repeatedly experimented with and sampled the original dataset five times [ 87 ]. Since comparing the results when the number of topics is 1 is not meaningful, it utilizes the perplexity scores from the second to the eighth topics of the original dataset as a baseline. These baseline scores are then compared against the perplexity scores generated by the subsets sampled on the other five occasions. A similarity test is subsequently conducted to assess the comparability shown in Fig 5 . The horizontal axis of the heat map above represents the number of topics using LDA for topic modeling, ranging from 2 to 8, and the vertical axis represents the number of subsets we randomly extracted from the original dataset [ 88 ]. Different squares represent the difference in perplexity between the topic modeling results of 5 randomly selected subsets of the original data set and the topic modeling of the original data set.

https://doi.org/10.1371/journal.pone.0302502.g005

In Fig 5 , the difference between the five tests and the baseline (perplexity of the original dataset) is small. When the number of topics is less than or equal to 6, the difference in confusion does not exceed 0.10. The relative maximum difference is when the number of topics in the second extracted subset is 8, at which time the difference in perplexity is only about 0.14. After several times of randomly extracting the subset and re-performing the number of topics-perplexity experiment, the perplexity difference is very small for five experiments in the range of the number of topics is 8, which further confirms the high robustness of our modeling. Rre-performing topic modeling on randomly selected subsets and topic modeling performed on the original dataset show extremely high similarity in results, and the results of multiple repetitions of the modeling show very little difference in perplexity. Therefore, based on the comparatively low perplexity differences between the subsets and the original data, the robustness of the model can be supported.

4.4 Sentiment analysis

In this part, two sentiment analysis models, Vader and Textblob, are assigned weights of 0.6 and 0.4 respectively for sentiment classification to explore research question 2. The sentiment analysis categorizes the emotional tone of the entries into three distinct parameters: positive, negative, and neutral. The weighting of positive, neutral, and negative entries is shown in Fig 6 (N = 23,773). The analysis reveals a positive sentiment among Reddit users, with approximately 61.6% of entries conveying affirmative emotional nuances. In contrast, about 20.8% of entries express negative sentiments, while neutral ones account for the smallest segment at 17.6%.

https://doi.org/10.1371/journal.pone.0302502.g006

Fig 7 demonstrates the comprehensive sentiment distribution (N = 23,773). There is a noticeable concentration of entries between 0 and 0.6, indicating the prevailing positive emotions. Moreover, it is pertinent to mention that the sentiment analysis identified a significant count exceeding 5100 entries conveying a neutral sentiment. Furthermore, most entries fall within the range of -0.25 to 0.60, suggesting a moderately nuanced sentiment orientation and a notable absence of distinct polarization in the overall sentiment attitudes towards ChatGPT.

https://doi.org/10.1371/journal.pone.0302502.g007

Table 5 selectively displays the high-frequency words in posts and comments expressing different sentiments.

https://doi.org/10.1371/journal.pone.0302502.t005

In neutral discussions, individuals mention the usage of ChatGPT, such as "api", and "to access it you have to use the API." The API is an interface that facilitates communication between distinct software systems or services, enabling programs or applications to access the functionalities or data of other systems. Users might engage in conversations regarding the functionalities and limitations of the API, deliberating on the prospect of integrating ChatGPT’s language generation capabilities into their applications or systems using the API [ 89 ].

The negative comments with words such as "wrong," "bad," and "problem," reflect their perception of errors, issues, or flaws of ChatGPT. For instance, "It is just a dumb stunt for a dumb application," "The model has been quantized badly, " and "I have doubts about their security claims. " These express suspicion about certain aspects of ChatGPT which generate problematic content at times or unsatisfactory functionality. It suggests skepticism about the accuracy and quality of the content generated by ChatGPT.

It is worth noting that the term "model" appears in both positive, neutral, and negative posts and comments, showcasing varying perspectives on the ChatGPT based on its performance, applications, and potential risks. Positive comments emphasize its impressive capabilities, including generating high-quality text, conducting fluent conversations, and efficiently retrieving information. This technology is acknowledged for its significant advances in natural language processing, benefiting various fields such as intelligent assistants and text creation. On the other hand, negative feedback may suggest that ChatGPT produces incorrect outputs, raises ethical concerns, and has the potential to spread misinformation. These diverse viewpoints reflect the complexity of ChatGPT and its social implications.

4.5 Sentiment trend analysis

This part examines daily sentiment trends by comparing the quantity of positive and negative sentiment posts from January to August 2023 (N = 23,773) to explore research question 3. Based on the GPT-3.5 model, ChatGPT was launched by Open AI on November 30, 2022, gaining a growing user base. On March 15, 2023, OpenAI unveiled the new multimodal model, GPT-4, available for purchase [ 2 ]. It aims to ascertain whether version updates have influenced sentiment towards ChatGPT. Fig 8 demonstrates how sentiments changed over time. The graph displays two sentiment classes, denoted by green and red, representing positive and negative sentiments, respectively. The sentiments fluctuate over time during the ChatGPT update.

https://doi.org/10.1371/journal.pone.0302502.g008

Fig 8 shows that fluctuations in 23,773 entries occur during mid-February, mid-to-late March, May, and mid-July. In most instances, the number of positive entries surpasses that of negative ones. The graph reveals a moderately increasing trend in mid-February, where the daily entry count exceeded 350. This surge can be attributed to the launch of the Plus plan on February 9, offering Plus users the option to select from various versions of ChatGPT. Moreover, the worldwide release of ChatGPT Plus for purchase was announced on February 13th. In mid-March, there was a small peak in the count of entries, approaching nearly 400 daily, perhaps attributable to the announcement of GPT-4 on March 14, 2023. Additionally, there were 100 more positive entries than negative ones indicating that GPT-4’s launch offered advanced reasoning, complex instructions, and enhanced creativity. In May, discussions regarding ChatGPT reached a peak. This surge in activity can be attributed to several factors. Primarily, it could be due to OpenAI’s implementation of new privacy features on May 3rd, which introduced the option to "Turn off chat history and decline to use for model training." It addresses some privacy concerns and encourages user engagement. Furthermore, on May 12th, OpenAI released an update allowing ChatGPT Plus members to incorporate the Bing search engine for browsing web content. It transcended the previous limitations of ChatGPT’s database, which had been confined to information available only until 2021. Lastly, it could also be largely attributed to the momentous launch of the ChatGPT iOS app on May 18th in selected countries, including the United States, the United Kingdom, France, and others. It enriches users’ mobile experiences. However, due to the significant limitations imposed by these successive updates on the nation and the mobile operating system, most users have been unable to benefit from the conveniences. This has potentially led to public dissatisfaction, resulting in a surge of more than 300 negative comments and posts in a single day.

In mid-July, there was a slight increase in ChatGPT discussions, with more than 300 entries reflecting positivity. The surge of positive entries can be attributed to the widespread introduction of the Code Interpreter feature to all ChatGPT Plus users. This innovation allowed non-programmers to express intentions in everyday language, translating into executable Python code solutions, enabling the accomplishment of intricate tasks within a real-time working environment. The innovation not only streamlined the processes of code composition and data manipulation but also expedited the application of artificial intelligence across diverse domains. While the version updates of ChatGPT may spark heated discussions among Reddit users, users generally hold positive sentiments towards ChatGPT and there was no significant shift from positive to negative attitudes during the period between January 2023 and August 2023.

5 Discussion

ChatGPT is one of the most fascinating frontier AI technologies, revolutionizing the approach to human-machine interaction and gaining worldwide attention for providing detailed answers in various areas of human society. However, there is an absence of studies evaluating its significant social influence. This study investigates the public’s viewpoints regarding the usage and impact of ChatGPT through topic modeling and sentiment analysis. Differing from sampling survey methods, this study follows the emerging trend of big data mining and gathers data on posts and comments from social media platform, Reddit. It employs the LDA unsupervised learning model to generate seven topics. The study uses a weighted approach that combines VADER with Textblob to categorize sentiment and analyze sentiment trends in posts and comments.

The result reveals seven topics of public discourse concerning ChatGPT, which can be classified into three themes: user perception, technical methods, and impacts on society. It suggests a comprehensive exploration by users into its potential ramifications, with opportunities for advancement across various facets of human society, such as markets, capital, employment, education, research, healthcare, art, entertainment, politics, gender, and ethical considerations. Meanwhile, the extensive discourse on its technical methods indicates that ChatGPT does not replace human intelligence or hinder creative expression. On the contrary, it provides a reservoir of diverse perspectives, facilitating unconventional thinking, and fostering an environment conducive to the expansion of human creative capacities [ 90 , 91 ].

In addition, sentiment analysis shows that people generally have a positive attitude towards ChatGPT. They believe that ChatGPT can engage in natural and easy conversations with users without requiring an in-depth understanding of complex natural language processing techniques. It is considered a symbol of huge technological progress. However, posts and comments still express concern and criticism about potential risks with ChatGPT. While there are acknowledged limitations within ChatGPT, this study does not explicitly pinpoint the specific areas where these problems exist. Finally, the sentiment analysis reveals that throughout the majority of the periods investigated in our study, most users express a positive attitude towards ChatGPT. Changes in sentiment tend to vary over time and may be affected by updates introduced to ChatGPT. These updates are often associated with a high level of user satisfaction on Reddit.

For practical implication, this study offers valuable insights into potential enhancements and optimal utilization strategies for developers and users of ChatGPT. GPT-related companies and developers should prioritize the user experience. While the public’s attitude towards it is relatively positive due to its naturalistic interactive capabilities, a substantial portion of public discourse (as one of the themes) concentrates on the technical methods of using ChatGPT and its prompts. Therefore, it is recommended that ChatGPT developers enhance the user-friendliness of bot features in product design and its prompt. Additionally, GPT-related companies and research institutions could consider prompt in-depth discussions on technological applications and impacts on society to attract more users. The application of ChatGPT in various fields, such as healthcare, art, and science, can encourage users to unlock the potential of ChatGPT. It promotes cross-domain integration and fosters innovation, even for those with limited knowledge of artificial intelligence techniques or programming [ 92 ]. Furthermore, by actively seeking dialogue from diverse stakeholders, this inclusive approach facilitates the ethical development and deployment of ChatGPT.

For the users, they should understand the impact of ChatGPT on their own lives and learn how to use it effectively. The general public needs to learn how to use suitable prompts for text generation and dialogue accurately. Also, users should consider the advantages and disadvantages of ChatGPT. Similar to the findings revealed by previous research [ 93 ], the public also expresses concerns about the ethical risks associated with ChatGPT, such as the potential for generating fabricated misinformation, violating copyrights, and promoting plagiarism. Therefore, all stakeholders are expected to cultivate social awareness and engage in public discourse regarding the ethical use and standards of technology. It is crucial to enhance the transparency, accountability, and fairness of ChatGPT [ 94 ].

Despite its contributions, this study has several limitations. First, it relies on data from a single social media platform, Reddit, where the users’ demographic skews towards being male, young, white, and highly educated (63% of Reddit users have a Bachelor’s degree or higher) [ 12 , 57 ]. Previous research indicates that individuals with higher educational attainment and younger age groups exhibit a greater understanding of ChatGPT. This may raise concerns about the generalizability of the findings to users of other social media platforms and the public [ 95 ]. Future research should examine the public’s attitude towards ChatGPT on various social media platforms to address the limitation. Comparative analyses across different platforms such as Twitter, Facebook, and online forums would provide a more comprehensive view and public perceptions of ChatGPT. Second, the study is descriptive, and future research should consider causal studies. The study shows a wide range of impacts of ChatGPT on different domains of human society (e.g., market, capital, employment, health, arts, entertainment, politics, and gender). However, it is uncertain whether users with different occupations and identities affect people’s attitudes toward ChatGPT. For example, quantitative methods such as regression analysis can be used. In addition, a longitudinal research design could explore how ChatGPT affects different domains over time. Third, this study does not identify the specific areas in which people expressed negative perceptions. A more detailed qualitative content analysis could examine negative posts and comments to identify specific themes and underlying concerns. This can lead to a better understanding of the limitations of the technology and directions for improvement.

Supporting information

S1 file. details on data collection and analysis..

https://doi.org/10.1371/journal.pone.0302502.s001

• View Article
• Google Scholar
• 6. Metz, C., & Collins, K. (2023, Mar 15). All the ways GPT-4 is impressive but still flawed: [Business/Financial desk]. New York. 2023. [cited 2023 December 2]. Available from: https://www.nytimes.com/2023/03/14/technology/openai-new-gpt4.html
• PubMed/NCBI
• 11. Brigham K. Reddit throughout the years: Its rise to prominence, recent revolts and IPO plans. CNBC. 2023. [cited 2023 December 2]. Available from: https://www.cnbc.com/2023/07/30/reddits-rise-to-prominence-recent-revolts-and-future-prospects.html .
• 47. Rajput NK, Grover BA, Rathi VK. Word frequency and sentiment analysis of Twitter messages during the coronavirus pandemic. arXiv:2004.03925 [Preprint]. 2020 [cited 2024 January 2]. Available from: https://doi.org/10.48550/arXiv.2004.03925 . https://doi.org/10.48550/arXiv.2004.03925
• 70. Anandarajan M, Hill C, Nolan T. Practical Text Analytics. Springer; 2018.

This paper is in the following e-collection/theme issue:

Published on 14.5.2024 in Vol 12 (2024)

The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation

Authors of this article:

• Yong Nam Gwon 1, * , MD ; 
• Jae Heon Kim 1, * , MD, PhD ; 
• Hyun Soo Chung 2 , MD ; 
• Eun Jee Jung 2 , MD ; 
• Joey Chun 1, 3 , MD ; 
• Serin Lee 1, 4 , MD ; 
• Sung Ryul Shim 5, 6 , MPH, PhD

1 Department of Urology, Soonchunhyang University College of Medicine, Soonchunhyang University Seoul Hospital, , Seoul, , Republic of Korea

2 College of Medicine, Soonchunhyang University, , Cheonan, , Republic of Korea

3 Cranbrook Kingswood Upper School, , Bloomfield Hills, MI, , United States

4 Department of Biochemistry, Case Western Reserve University, , Cleveland, OH, , United States

5 Department of Biomedical Informatics, Konyang University College of Medicine, , Daejeon, , Republic of Korea

6 Konyang Medical Data Research Group-KYMERA, Konyang University Hospital, , Daejeon, , Republic of Korea

*these authors contributed equally

Corresponding Author:

Sung Ryul Shim, MPH, PhD

Background: A large language model is a type of artificial intelligence (AI) model that opens up great possibilities for health care practice, research, and education, although scholars have emphasized the need to proactively address the issue of unvalidated and inaccurate information regarding its use. One of the best-known large language models is ChatGPT (OpenAI). It is believed to be of great help to medical research, as it facilitates more efficient data set analysis, code generation, and literature review, allowing researchers to focus on experimental design as well as drug discovery and development.

Objective: This study aims to explore the potential of ChatGPT as a real-time literature search tool for systematic reviews and clinical decision support systems, to enhance their efficiency and accuracy in health care settings.

Methods: The search results of a published systematic review by human experts on the treatment of Peyronie disease were selected as a benchmark, and the literature search formula of the study was applied to ChatGPT and Microsoft Bing AI as a comparison to human researchers. Peyronie disease typically presents with discomfort, curvature, or deformity of the penis in association with palpable plaques and erectile dysfunction. To evaluate the quality of individual studies derived from AI answers, we created a structured rating system based on bibliographic information related to the publications. We classified its answers into 4 grades if the title existed: A, B, C, and F. No grade was given for a fake title or no answer.

Results: From ChatGPT, 7 (0.5%) out of 1287 identified studies were directly relevant, whereas Bing AI resulted in 19 (40%) relevant studies out of 48, compared to the human benchmark of 24 studies. In the qualitative evaluation, ChatGPT had 7 grade A, 18 grade B, 167 grade C, and 211 grade F studies, and Bing AI had 19 grade A and 28 grade C studies.

Conclusions: This is the first study to compare AI and conventional human systematic review methods as a real-time literature collection tool for evidence-based medicine. The results suggest that the use of ChatGPT as a tool for real-time evidence generation is not yet accurate and feasible. Therefore, researchers should be cautious about using such AI. The limitations of this study using the generative pre-trained transformer model are that the search for research topics was not diverse and that it did not prevent the hallucination of generative AI. However, this study will serve as a standard for future studies by providing an index to verify the reliability and consistency of generative AI from a user’s point of view. If the reliability and consistency of AI literature search services are verified, then the use of these technologies will help medical research greatly.

Introduction

The global artificial intelligence (AI) health care market size was estimated to be at US $15.1 billion in 2022 and is expected to surpass approximately US $187.95 billion by 2030, growing at an annualized rate of 37% during the forecast period from 2022 to 2030 [ 1 ]. In particular, innovative applications of medical AI are expected to increase in response to medical demand, which will explode in 2030 [ 2 , 3 ].

A large language model (LLM) is a type of AI model that opens up great possibilities for health care practice, research, and education, although scholars have emphasized the need to proactively address the issue of unvalidated and inaccurate information regarding its use [ 4 , 5 ]. One of the best-known LLMs is ChatGPT (OpenAI). It was launched in November 2022. Similar to other LLMs, ChatGPT is trained on huge text data sets in numerous languages, allowing it to respond to text input with humanlike responses [ 4 ]. Developed by the San Francisco–based AI research laboratory OpenAI, ChatGPT is based on a generative pre-trained transformer (GPT) architecture. It is considered an advanced form of a chatbot, an umbrella term for a program that uses a text-based interface to understand and generate responses. The key difference between a chatbot and ChatGPT is that a chatbot is usually programmed with a limited number of responses, whereas ChatGPT can produce personalized responses according to the conversation [ 4 , 6 ].

Sallam’s [ 5 ] systematic review (SR) sought to identify the benefits and current concerns regarding ChatGPT. That review advises that health care research could benefit from ChatGPT, since it could be used to facilitate more efficient data set analysis, code generation, and literature reviews, thus allowing researchers to concentrate on experimental design as well as drug discovery and development. The author also suggests that ChatGPT could be used to improve research equity and versatility in addition to its ability to improve scientific writing. Health care practice could also benefit from ChatGPT in multiple ways, including enabling improved health literacy and delivery of more personalized medical care, improved documentation, workflow streamlining, and cost savings. Health care education could also use ChatGPT to provide more personalized learning with a particular focus on problem-solving and critical thinking skills [ 5 ]. However, the same review also lays out the current concerns, including copyright issues, incorrect citations, and increased risk of plagiarism, as well as inaccurate content, risk of excessive information leading to an infodemic on a particular topic, and cybersecurity issues [ 5 ].

A key question regarding the use of ChatGPT is if it can use evidence to identify premedical content. Evidence-based medicine (EBM) provides the highest level of evidence in medical treatment by integrating clinician experience, patient value, and best-available scientific information to guide decision-making on clinical management [ 7 ]. The principle of EBM means that the most appropriate treatment plan for patients should be devised based on the latest empirical research evidence. However, the scientific information identified by ChatGPT is not yet validated in terms of safety or accuracy according to Sallam [ 5 ], who further suggests that neither doctors nor patients should rely on it at this stage. In contrast, another study by Zhou et al [ 8 ] found that answers provided by ChatGPT were generally based on the latest verified scientific evidence, that is, the advice given followed high-quality treatment protocols and adhered to guidelines from experts.

In medicine, a clinical decision support system (CDSS) uses real-time evidence to support clinical decision-making. This is a fundamental tool in EBM, which uses SRs based on a systematic, scientific search of a particular subject. If ChatGPT becomes a CDSS, it is fundamental to determine whether it is capable of performing a systematic search based on real-time generation of evidence in the medical field. Therefore, this study will be the first to determine whether ChatGPT can search papers for an SR. In particular, this study aims to present a standard for medical research using generative AI search technology in the future by providing indicators for the reliability and consistency of generative AI searches from a user’s perspective.

Ethical Considerations

As per 45 CFR §46.102(f), the activities performed herein were considered exempt from institutional review board approval due to the data being publicly available. Informed consent was not obtained, since this study used previously published deidentified information that was available to the general public. This study used publicly available data from PubMed, Embase, and Cochrane Library and did not include human participant research.

Setting the Benchmark

To determine whether ChatGPT, currently the most representative LLM, is capable of systematic searches, we set an SR that was performed by human experts as a benchmark and checked how many studies were finally included in the benchmark were presented by ChatGPT. We chose Lee et al [ 9 ] as the benchmark for the following reasons. First, Lee et al [ 9 ] performed an SR and meta-analysis about the medical treatment for Peyronie disease (PD) with human experts. PD typically presents with discomfort, curvature, or deformity of the penis in association with palpable plaques and erectile dysfunction [ 10 ]. Second, it was easy to compare the results of ChatGPT and the benchmark, because we had full information about the interim process and results of the study. Third, a sufficient amount of studies has been published about the medical treatment for PD, but there is still no consensus answer. So, we expected to assess the sole ability of ChatGPT as a systematic search tool with sufficient data while avoiding any possible pretrained bias. Lastly, with the topic of Lee et al [ 9 ], we could build questions that start broad and become more specific and add some conditions that could test ChatGPT’s comprehension about scientific research. For example, questions could not only be built broadly by asking about “medical treatment for Peyronie’s disease” but also specifically by asking about “oral therapy for Peyronie’s disease” or “colchicine for Peyronie’s disease.” Because Lee et al [ 9 ] only contained randomized controlled trials (RCTs), we could add a condition to the questions to restrict the study type to RCTs, which could be useful to assess the comprehension of ChatGPT.

Systematic Search Formula of Benchmark

Lee et al [ 9 ] used the following search query in PubMed and Cochrane Library: (“penile induration”[MeSH Terms] OR “Peyronie’s disease”[Title/Abstract]) AND “male”[MeSH Terms] AND “randomized controlled trial”[Publication Type] , and the following query in Embase: (‘Peyronie disease’/exp OR ’Peyronie’s diseas’:ab,ti) AND ’male’/exp AND ’randomized controlled trial’/de . After the systematic search, a total of 217 records were identified. Studies were excluded for the following reasons: not RCTs, not perfectly fit to the topic, not enough sample size or outcome, and not written in English. Finally, 24 RCTs were included in the SR, with only 1 RCT published in 2022 ( Figure 1 ) [ 9 ]. The characteristics of all studies included in Lee et al [ 9 ] are summarized in Section S1 in Multimedia Appendix 1 .

Methodology of Systematic Search for ChatGPT

Based on the search formula used in Lee et al [ 9 ], a simple mandatory prompt in the form of a question was created, starting with comprehensive questions and gradually asking more specific questions ( Textbox 1 ). For example, questions could be built as “Could you show RCTs of colchicine for Peyronie’s disease in PubMed?” with the treatment and database changed under the same format. In addition to mandatory questions, we added questions about treatment additionally provided by ChatGPT during the conversation. Considering the possibility that ChatGPT might respond differently depending on the interaction, we arranged questions into 2 logical flows, focusing on database and treatment, respectively ( Figure 2 and Figure S1 in Multimedia Appendix 1 ). We asked about search results from 4 databases: PubMed [ 11 ], Google (Google Scholar) [ 12 ], Cochrane Library [ 13 ], and ClinicalTrials.gov [ 14 ]. PubMed is a leading biomedical database offering access to peer-reviewed articles. Google Scholar provides a wide-ranging index of scholarly literature, including medical studies. Cochrane Library specializes in high-quality evidence through SRs and clinical trials. ClinicalTrials.gov, managed by the National Library of Medicine, serves as a comprehensive repository for clinical study information globally. These databases collectively serve researchers by providing access to diverse and credible sources, facilitating literature reviews and evidence synthesis, and informing EBM in the medical field. They play crucial roles in advancing medical knowledge, supporting informed decision-making, and ultimately improving patient care outcomes [ 11 - 14 ]. These 4 databases were easy to access and contained most of the accessible studies. Each question was repeated at least twice. We extracted the answers and evaluated the quality of information based on the title, author, journal, and publication year (Sections S2-S5 Multimedia Appendix 1 ).

Basic format of questions

• “Could you show RCTs of (A) for Peyronie’s disease in (B)?”

(A) Treatment category and specific treatment

• Vitamin E, colchicine, L-carnitine, potassium aminobenzoate, tamoxifen, pentoxifylline, tadalafil, L-arginine, and sildenafil
• Verapamil, interferon-a2B, collagenase Clostridium histolyticum , transdermal electromotive administration, hyaluronidase, triamcinolone, mitomycin C, super-oxide dismutase, and 5-fluorouracil
• Extracorporeal shockwave therapy, iontophoresis, traction therapy, vacuum, penile massage, and exercise shockwave therapy
• 5-Alpha-reductase inhibitors, superficial heat, diclofenac gel, collagenase Clostridium histolyticum gel, verapamil gel, potassium aminobenzoate gel, and propionyl-L-carnitine gel

(B) Database

• Google (Google Scholar)
• Cochrane Library
• ClinicalTrials.gov

We used the GPT-3.5 version of ChatGPT, which was pretrained with data before 2021, for the systematic search and evaluated how many RCTs that were included in Lee et al [ 9 ] were present in the search results from ChatGPT. To assess the reliability of ChatGPT’s answers, we also evaluated whether the studies presented actually existed. ChatGPT’s response style and the amount of information might vary from answer to answer. Thus, we evaluated the accuracy of the responses by prioritizing a match by (1) title; (2) author, journal, and publication year; and (3) other items.

To obtain higher-quality responses, it is important to structure the prompts using refined language that is well understood by the LLM [ 15 - 17 ]. In this study, we performed the following fine-tuning to clearly convey the most important content or information. We first defined roles and provided context and input data before asking complete questions to get responses, and we used specific and varied examples to help the model narrow its focus and produce more accurate results [ 18 , 19 ]. During the prompt engineering, the treatment category, specific treatment, and target databases were structured in order, and the order was changed in the detailed elements to induce consistent answers. Details of this are presented in Multimedia Appendix 1 .

Quality Assessment of Answers

To evaluate the quality of individual studies derived from AI answers, we created a structured rating system based on bibliographic information related to the publications ( Table 1 ). We classified its answers into 4 grades if the title existed: A, B, C, and F. No grade was given for a fake title or no answer.

a PICOS: population, intervention, comparison, outcome, and study design (research questions).

c Any mismatch in essential information or accessory information.

d Mismatch.

e N/A: not assessed.

A grade of “A” was given to an answer that was appropriate for the question and perfectly consistent with the actual study. For example, for the question “Could you show all RCTs of medical treatment for Peyronie’s disease in Google?” ChatGPT answered “Gelbard MK, James K, Riach P, Dorey F. Collagenase versus placebo in the treatment of Peyronie’s disease: a double-blind study. J Urol. 1993; 149(1):56-58.” This answer was perfectly consistent with actual study because the authors, title, journal, publication year, issue number, and even page number matched, and it was appropriate because it was an RCT, was about medical treatment for PD, and could be found in Google.

A grade of “B” was given to an answer that was appropriate for the question and the title, author, journal, and publication year matched with the actual study. For example, for the question “Could you show RCTs of Intralesional therapy for Peyronie’s disease in Cochrane Library?” ChatGPT answered “Gelbard MK, James K, Riach P, Dorey F. Collagenase versus placebo in the treatment of Peyronie’s disease: a double-blind study. J Urol. 1993 Feb; 149(2):56-8. doi: 10.1016/s0022-5347(17)36148-1. PMID: 8426418.” This answer was appropriate because it was an RCT, was about intralesional therapy for PD, and could be found in Cochrane Library. However, this answer was consistent with the title, authors, journal, and publication year of actual study but inconsistent with the accessory information for publication month, issue number, page number, DOI, and PMID. So, it was able to specify the presented study with essential information, but the accessory information was not consistent with the actual study.

A grade of “C” was given to an answer that was inappropriate for the question but the title, author, journal, and publication year matched with the actual study. For example, for the question “Could you show RCTs of Pentoxifylline for Peyronie’s disease in PubMed?” ChatGPT answered “Gelbard MK, James K, Riach P, Dorey F. Collagenase versus placebo in the treatment of Peyronie’s disease: a double-blind study. J Urol. 1993; 149(1):56-8. doi: 10.1016/s0022-5347(17)36091-7. PMID: 8417228.” This answer was consistent with the title, authors, journal, and publication year of the actual study, but it was inappropriate because it was not about the use of pentoxifylline for PD.

A grade of “F” was given to an answer that was inconsistent with the author, journal, or publication year of the actual study that matched the title, making it impossible to be specified. Thus, it was considered as a fake study that was generated. For example, for the question “Could you show RCTs of collagenase Clostridium histolyticum for Peyronie’s disease in PubMed?” ChatGPT answered “Gelbard MK, James K, Riach P, Dorey FJ, & Collagenase Study Group. (2012). Collagenase versus placebo in the treatment of Peyronie’s disease: a double-blind study. The Journal of urology, 187(3), 948-953.” This answer was consistent with the title of the actual study but inconsistent with the authors, publication year, and so on.

Searching Strategy for Bing AI

To compare with ChatGPT, we performed the same process with Bing AI [ 20 ], also known as “New Bing,” an AI chatbot developed by Microsoft and released in 2023. Since Bing AI functions based on the huge AI model “Prometheus” that includes OpenAI’s GPT-4 with web searching capabilities, it is expected to give more accurate answers than the GPT-3.5 version of ChatGPT. We performed the conversation with the “Precise” tone. Because Bing AI limited the number of questions per session to 20, we did not arrange questions into 2 logical flows (Section S6 in Multimedia Appendix 1 ). We compared the number of studies included in the benchmark [ 9 ] and provided by Bing AI. We also evaluated the reliability of answers with the same method described above or using links of websites presented by Bing AI (Figure S2 and Section S7 in Multimedia Appendix 1 ).

Systematic Search Results via ChatGPT

A total of 639 questions were entered into ChatGPT, and 1287 studies were obtained ( Table 2 ). The systematic search via ChatGPT was performed from April 17 to May 6, 2023. At the beginning of the conversation, we gave ChatGPT the role of a researcher conducting a systematic search who intended to perform a meta-analysis for more appropriate answers. At first, we tried to build question format by using the word “find,” such as “Could you find RCTs of medical treatment for Peyronie’s disease?” However, ChatGPT did not present studies and only suggested how to find RCTs in a database, such as PubMed. Therefore, we changed the word “find” to “show,” and ChatGPT presented lists of RCTs. For comprehensive questions, ChatGPT did not give an answer, saying that it did not have the capability to show a list of RCTs as an AI language model. However, when questions were gradually specified, it created answers (Sections S2 and S4 in Multimedia Appendix 1 ).

a AI: artificial intelligence.

b From Lee et al [ 9 ].

Of the 1287 studies provided by ChatGPT, only 7 (0.5%) studies were perfectly eligible and 18 (1.4%) studies could be considered suitable under the assumption that they were real studies if only the title, author, journal, and publication year matched ( Table 2 ). Among these, only 1 study was perfectly consistent with studies finally included in Lee et al [ 9 ], and 4 studies were matched under the assumption (Sections S1, S3, and S5 in Multimedia Appendix 1 ).

Specifically, systematic search via ChatGPT was performed in 2 logical flow schemes, database setting and treatment setting ( Figure 2 and Figure S1 in Multimedia Appendix 1 ). With the logical flow by database setting, among the 725 obtained studies, 6 (0.8%) and 8 (1.1%) studies were classified as grade A and grade B, respectively ( Table 1 ). Of these, 1 grade A study and 1 grade B study were included in Lee et al [ 5 ]. With the logical flow by treatment setting, among the 562 obtained studies, 1 (0.2%) study was classified as grade A and 10 (1.8%) studies were classified as grade B. Of these, 3 grade B studies were included in the benchmark [ 9 ] ( Table 2 ).

It was common for answers to be changed. There were many cases where answers contradicted themselves. In addition, there were cases where the answer was “no capability” or “no RCT found” at first, but when another question was asked and the previous question was asked again, an answer was given. ChatGPT showed a tendency to create articles by rotating some format and words. Titles presented were so plausible that it was almost impossible to identify fake articles until an actual search was conducted. The presented authors were also real people. Titles often contained highly specific numbers, devices, or brand names that were real. There were some cases where it was possible to infer which articles ChatGPT mimicked in the fake answers (Sections S3 and S5 in Multimedia Appendix 1 ). Considering these characteristics, when generating sentences, ChatGPT seemed to list words with a high probability of appearing among pretrained data rather than presenting accurate facts or understanding questions.

In conclusion, of the 1287 studies presented by ChatGPT, only 1 (0.08%) RCT matched the 24 RCTs of the benchmark [ 9 ].

Systematic Search Results via Bing AI

For Bing AI, a total of 223 questions were asked and 48 studies were presented. Among the 48 obtained studies, 19 (40%) studies were classified as grade A. There were no grade B studies ( Table 2 ). Because Bing AI always gave references with links to the websites, all studies presented by Bing AI existed. However, it also provided wrong answers about the study type, especially as it listed reviews as RCTs. Of the 28 studies with grade C, 27 (96%) were not RCTs and 1 (4%) was about a different treatment. Only 1 study had no grade because of a fake title; it presented a study registered in PubMed while pretending that it was the result of a search in ClinicalTrials.gov. However, the study was not in ClinicalTrials.gov (Section S7 in Multimedia Appendix 1 ).

Bing AI had more accurate answers than ChatGPT since it provides actual website references. However, it also showed a tendency to give more answers to more specific questions, similar to ChatGPT. For example, with a comprehensive question, Bing AI said “I am not able to access or search specific databases.” However, with more specific questions, it found studies or answered “I couldn’t find any RCTs’ without mention about accessibility.” In most cases, Bing AI either failed to find studies or listed too few studies to be used as a systematic searching tool.

In conclusion, of the 48 studies presented by Bing AI, 2 (4%) RCTs matched the 24 RCTs of the benchmark [ 9 ].

Principal Findings

This paper’s researchers sought to determine whether ChatGPT could conduct a real-time systematic search for EBM. For the first time, researchers compared the performance of ChatGPT with classic systematic searching as well as the Microsoft Bing AI search engine. Although Zhou et al [ 8 ] suggested that ChatGPT answered qualitative questions based on recent evidence, this study found that ChatGPT’s results were not based on a systematic search (which is the basis for an SR), meaning that they could not be used for real-time CDSS in their current state.

With recent controversy regarding the risks and benefits of advanced AI technologies [ 21 - 24 ], ChatGPT has received mixed responses from the scientific community and academia. Although many scholars agree that ChatGPT can increase the efficiency and accuracy of the output in writing and conversational tasks [ 25 ], others suggest that the data sets used in ChatGPT’s training might lead to possible bias, which not only limits its capabilities but also leads to the phenomenon of hallucination—apparently scientifically plausible yet factually inaccurate information [ 24 ]. Caution around the use of LLMs should also bear in mind security concerns, including the potential of cyberattacks that deliberately spread misinformation [ 25 ].

When applying the plug-in method in this study, especially when using PubMed Research [ 26 ], the process worked smoothly and there was not a single case of hallucination of fake research (by providing information along with a link), regardless of the designation of a specific database engine. Among the responses, 21 RCTs were included in the final SR, and out of a total of 24, all RCTs except 3 were provided. This is a very encouraging result. However, there is no plug-in that allows access to other databases yet, and if the conversation is long, the response speed is very slow. Furthermore, although it is a paid service, it only provides a total of 100 papers, so if more than 100 RCTs are searched, the user must manually search all papers. Ultimately, it is not intended for conducting an efficient and systematic search, as additional time and effort are required. If a more efficient plug-in is developed, this could play a promising part in systematic searches.

Although Sallam’s [ 5 ] SR suggests that academic and scientific writing as well as health care practice, research, and education could benefit from the use of ChatGPT, this study found that ChatGPT could not search scientific articles properly, with a 0.08% (1/1287) of probability of the desired paper being presented. In the case of Bing AI using GPT-4, this study showed that Bing AI could search scientific articles with a much higher accuracy than ChatGPT. However, the probability was only 4% (2/48). It was still an insufficient probability for performing systematic research. Moreover, fake answers generated by ChatGPT, known as hallucinations, caused researchers to spend extra time and effort by checking the accuracy of the answers. A typical problem with generative AI is that it creates hallucinations. However, this is difficult to completely remove due to the principle of generative AI. Therefore, if it cannot be prevented from the pretraining of the model, efforts to increase reliability and consistency in the use of generative AI in medical care by checking the accuracy from the user’s point of view are required, as shown in this study. Unlike ChatGPT, Bing AI did not generate fake studies. However, the total number of studies presented was too small. Very few studies have focused on the scientific searching accuracy of ChatGPT. Although this paper found many articles about the use of ChatGPT in the medical field, the majority concerned the role of ChatGPT as an author. Although the latter might accelerate writing efficiency, it also confirms the previously mentioned issues of transparency and plagiarism.

Wang et al [ 27 ] have recently investigated whether ChatGPT could be used to generate effective Boolean queries for an SR literature search. The authors suggest that ChatGPT should be considered a “valuable tool” for researchers conducting SRs, especially for time-constrained rapid reviews where trading off higher precision for lower recall is generally acceptable. They cite its ability to follow complex instructions and generate high-precision queries. Nonetheless, it should be noted that building a Boolean query is not a complex process. However, selecting the most appropriate articles for an SR is critical, which might be a more useful subject to examine in relation to the use of ChatGPT. Moreover, although Aydın and Karaarslan [ 28 ] have indicated that ChatGPT shows promise in generating a literature review, the iThenticate plagiarism tool found significant matches in paraphrased elements.

In scientific research, the most time-consuming and challenging task can be the process of filtering out unnecessary papers on the one hand and identifying those that are needed on the other hand. This difficult yet critical task can be daunting. It discourages many researchers from participating in scientific research. If AI could replace this process, it will be easier to collect and analyze data from the selected papers. Recently, commercial literature search services using generative AI models have emerged. Representative examples include Covidence [ 29 ], Consensus [ 30 ], and Elicit [ 31 ]. The technical details of these commercial AI literature search services are unknown, but they are based on LLMs using GPT. Therefore, these search services are not only insufficient to verify hallucinations but also lack information in the search target databases. Even if there may be mistakes, the researcher should aim for completeness, and unverified methods should be avoided. Although this study did not use a commercial literature search service, it manually searched the target databases one by one. If the reliability and consistency of AI literature search services are verified, the use of these technologies will help medical research greatly

This study suggests that ChatGPT still has limitations in academic search, despite the recent assertion from Zhou et al [ 8 ] about its potential in searching for academic evidence. Moreover, although ChatGPT can search and identify guidance in open-access guidelines, its results are brief and fragmentary, often with just 1 or 2 sentences that lack relevant details about the guidelines.

Arguably, more concern should be placed on the potential use of ChatGPT in a CDSS than its role in education or writing draft papers. On the one hand, if AI such as ChatGPT is used within a patient-physician relationship, this is unlikely to affect liability since the advice is filtered through professionals’ judgment and inaccurate advice generated by AI is no different from erroneous or harmful information disseminated by a professional. However, ChatGPT lacks sufficient accuracy and speed to be used in this manner. On the other hand, ChatGPT could also be used to give direct-to-consumer advice, which is largely unregulated since asking AI directly for medical advice or emotional support acts outside the established patient-physician relationship [ 32 ]. Since there is a risk of patient knowing inaccurate information, the medical establishment should seek to educate patients and guardians about the risk of inaccurate information in this regard.

Academic interest in ChatGPT to date has mainly focused on potential benefits including research efficiency and education, drawbacks related to ethical issues such as plagiarism and the risk of bias, as well as security issues including data privacy. However, in terms of providing medical information and acting as a CDSS, the use of ChatGPT is currently less certain because its academic search capability is potentially inaccurate, which is a fundamental issue that must be addressed.

The limitation of this study is that it did not address various research topics, because only 1 research topic was searched when collecting target literature. In addition, due to the time difference between the start of the study and the review and evaluation period, the latest technology could not be fully applied because it could become an outdated technology in a field of study where technology advances rapidly, such as generative AI. For example, there have already been significant technological advances since new AI models such as ChatGPT Turbo (4.0) were released between the time we started this study and the current revised time point.

This paper thus suggests that the use of AI as a tool for generating real-time evidence for a CDSS is a dream that has not yet become a reality. The starting point of evidence generation is a systematic search and ChatGPT is unsuccessful even for this initial purpose. Furthermore, its potential use in providing advice directly to patients in a direct-to-consumer form is concerning, since ChatGPT could provide inaccurate medical information that is not evidence based and can result in harm. For the proper use of generative AI in medical care in the future, it is suggested that a feedback model that evaluates accuracy according to experts’ perspective, as done in this study, and then reflects it back into an LLM is necessary.

This is the first study to compare AI and conventional human SR methods as a real-time literature collection tool for EBM. The results suggest that the use of ChatGPT as a tool for real-time evidence generation is not yet accurate and feasible. Therefore, researchers should be cautious about using such AI. The limitations of this study using the GPT model are that the search for research topics was not diverse and that it did not prevent the hallucinations of generative AI. However, this study will serve as a standard for future studies by providing an index to verify the reliability and consistency of generative AI from a user’s point of view. If the reliability and consistency of AI literature search services are verified, the use of these technologies will help medical research greatly.

Acknowledgments

This work was supported by the Soonchunhyang University Research Fund. This body had no involvement in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.

Authors' Contributions

SRS had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. YNG, HSC, EJJ, JC, SL, and SRS contributed to the analysis and interpretation of data. YNG, HSC, SRS, and JHK contributed to the drafting of the manuscript. SRS and JHK contributed to critical revision of the manuscript for important intellectual content. YNG and SRS contributed to statistical analysis.

Conflicts of Interest

None declared.

Additional logical flow diagrams, characteristics of studies included in Lee et al [ 9 ], ChatGPT and Microsoft Bing transcripts, and grade classification for answers.

• Artificial intelligence (AI) in healthcare market (by component: software, hardware, services; by application: virtual assistants, diagnosis, robot assisted surgery, clinical trials, wearable, others; by technology: machine learning, natural language processing, context-aware computing, computer vision; by end user) - global industry analysis, size, share, growth, trends, regional outlook, and forecast 2022-2030. Precedence Research. Feb 2023. URL: https://www.precedenceresearch.com/artificial-intelligence-in-healthcare-market [Accessed 2024-03-31]
• Bajwa J, Munir U, Nori A, Williams B. Artificial intelligence in healthcare: transforming the practice of medicine. Future Healthc J. Jul 2021;8(2):e188-e194. [ CrossRef ] [ Medline ]
• Zahlan A, Ranjan RP, Hayes D. Artificial intelligence innovation in healthcare: literature review, exploratory analysis, and future research. Technol Soc. Aug 2023;74:102321. [ CrossRef ]
• Models. OpenAI. URL: https://platform.openai.com/docs/models/gpt-3-5 [Accessed 2023-06-14]
• Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). Mar 19, 2023;11(6):887. [ CrossRef ] [ Medline ]
• Brown TB, Mann B, Ryder N, et al. Language models are few-shot learners. arXiv. Preprint posted online on Jul 22, 2020. [ CrossRef ]
• Evidence-Based Medicine Working Group. Evidence-based medicine. a new approach to teaching the practice of medicine. JAMA. Nov 4, 1992;268(17):2420-2425. [ CrossRef ] [ Medline ]
• Zhou Z, Wang X, Li X, Liao L. Is ChatGPT an evidence-based doctor? Eur Urol. Sep 2023;84(3):355-356. [ CrossRef ] [ Medline ]
• Lee HY, Pyun JH, Shim SR, Kim JH. Medical treatment for Peyronie’s disease: systematic review and network Bayesian meta-analysis. World J Mens Health. Jan 2024;42(1):133. [ CrossRef ]
• Chung E, Ralph D, Kagioglu A, et al. Evidence-based management guidelines on Peyronie's disease. J Sex Med. Jun 2016;13(6):905-923. [ CrossRef ] [ Medline ]
• PubMed. URL: https://pubmed.ncbi.nlm.nih.gov/about/ [Accessed 2023-06-14]
• Google Scholar. URL: https://scholar.google.com/ [Accessed 2023-06-14]
• Cochrane Library. URL: https://www.cochranelibrary.com/ [Accessed 2023-06-14]
• ClinicalTrials.gov. URL: https://classic.clinicaltrials.gov/ [Accessed 2023-06-14]
• Nori H, Lee YT, Zhang S, et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv. Preprint posted online on Nov 28, 2023. [ CrossRef ]
• Ziegler A, Berryman J. A developer’s guide to prompt engineering and LLMs. GitHub Blog. Jul 17, 2023. URL: https://github.blog/2023-07-17-prompt-engineering-guide-generative-ai-llms/ [Accessed 2023-07-17]
• Introducing ChatGPT. OpenAI. Nov 30, 2022. URL: https://openai.com/blog/chatgpt [Accessed 2023-10-16]
• Reid R. How to write an effective GPT-3 or GPT-4 prompt. Zapier. Aug 3, 2023. URL: https://zapier.com/blog/gpt-prompt/ [Accessed 2023-10-14]
• Prompt engineering for generative AI. Google. Aug 8, 2023. URL: https://developers.google.com/machine-learning/resources/prompt-eng?hl=en [Accessed 2024-04-23]
• Bing. URL: https://www.bing.com/ [Accessed 2024-04-30]
• de Angelis L, Baglivo F, Arzilli G, et al. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health. Apr 25, 2023;11:1166120. [ CrossRef ] [ Medline ]
• Howard J. Artificial intelligence: implications for the future of work. Am J Ind Med. Nov 2019;62(11):917-926. [ CrossRef ] [ Medline ]
• Tai MCT. The impact of artificial intelligence on human society and bioethics. Tzu Chi Med J. Aug 14, 2020;32(4):339-343. [ CrossRef ] [ Medline ]
• Wogu IAP, Olu-Owolabi FE, Assibong PA, et al. Artificial intelligence, alienation and ontological problems of other minds: a critical investigation into the future of man and machines. Presented at: 2017 International Conference on Computing Networking and Informatics (ICCNI); Oct 29 to 31, 2017:;1-10; Lagos, Nigeria. [ CrossRef ]
• Deng J, Lin Y. The benefits and challenges of ChatGPT: an overview. Frontiers in Computing and Intelligent Systems. Jan 5, 2023;2(2):81-83. [ CrossRef ]
• PubMed Research. whatplugin.ai. URL: https://www.whatplugin.ai/plugins/pubmed-research [Accessed 2024-04-30]
• Wang S, Scells H, Koopman B, Zuccon G. Can ChatGPT write a good Boolean query for systematic review literature search? arXiv. Preprint posted online on Feb 9, 2023. [ CrossRef ]
• Aydın Ö, Karaarslan E. OpenAI ChatGPT generated literature review: digital twin in healthcare. In: Aydın Ö, editor. Emerging Computer Technologies 2. İzmir Akademi Dernegi; 2022;22-31. [ CrossRef ]
• Covidence. URL: https://www.covidence.org/ [Accessed 2024-04-24]
• Consensus. URL: https://consensus.app/ [Accessed 2024-04-24]
• Elicit. URL: https://elicit.com/ [Accessed 2024-04-24]
• Haupt CE, Marks M. AI-generated medical advice-GPT and beyond. JAMA. Apr 25, 2023;329(16):1349-1350. [ CrossRef ] [ Medline ]

Abbreviations

Edited by Alexandre Castonguay; submitted 24.07.23; peer-reviewed by In Gab Jeong Jeong, Jinwon Noh, Lingxuan Zhu, Sachin Pandey, Taeho Greg Rhee; final revised version received 31.03.24; accepted 04.04.24; published 14.05.24.

© Yong Nam Gwon, Jae Heon Kim, Hyun Soo Chung, Eun Jee Jung, Joey Chun, Serin Lee, Sung Ryul Shim. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 14.5.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/ , as well as this copyright and license information must be included.

ChatGPT in Higher Education: Facts and Ideas to Consider

Introduction

It is safe to say that by now, you have seen many articles/posts, opinions, and stories about ChatGPT—and the larger AI-Language Learning Models (LLMs)—in relation to higher education and teaching/learning in particular. These writings included several perspectives ranging from raising concerns to celebrating new opportunities and a mix of the former and the latter. Also, these writings continue to evolve and grow rapidly in number as new AI-powered LLMs continue to emerge and evolve (e.g., Google’s new AI LLMs: Bard ).

The intent of this piece is not to add another article sharing tips or concerns about ChatGPT. That being said, this article (1) summarizes the major concerns about ChatGPT and (2) the ideas about its positive implications based on what it is published to date.

Concerns about ChatGPT

Faculty, scholars, and higher education leaders have raised several concerns about ChatGPT. These concerns stem from possible ways it can be used.

• Using ChatGPT to cheat by asking it to write essays/answer open-ended questions in exams/discussion forums and homework assignments (December 19th, 2022 NPR Story ) (December 6th, 2022 Atlantic Story ) (January 16 New York Times Story ).
• Using ChatGPT to author scholarly works which conflict with the ethical standards of scientific inquiry. Several high-impact/profile journals have already formulated principles to guide authors on how to use LLMs AI tools and why it is not allowed to credit such tool as an author—any attribution of authorship carries with it accountability for the scholarly work, and no AI tool can take such responsibility (January 24th, 2023 Nature Editorial ).
• ChatGPT can threaten the privacy of students/faculty (and any other user). Its privacy policy states that data can be shared with third-party vendors, law enforcement, affiliates, and other users. Also, while one can delete their ChatGPT account, the prompts they entered into ChatGPT cannot be deleted. This setup threatens sensitive or controversial topics as this data cannot be removed (January 2023 Publication by Dr. Torrey Trust ).
• ChatGPT is not always trustworthy, as it can fabricate quotes and references. In an experiment conducted by Dr. Daniel Hickey at Indiana University Bloomington, Instructional Systems Technology department, “ ChatGPT was able to write a marginally acceptable literature review paper, but fabricated some quotes and references. With more work such as including paper abstracts in the prompts, GPT is scarily good at referencing research literature, perhaps as well as a first-year graduate student.” (January 6th, 2023, Article by Dr. Daniel Hickey )

Excitement about ChatGPT

At the other end of the spectrum, there have been several ideas that express interest and excitement about ChatGPT in higher education. These ideas stem from how they can be used ethically and in a controlled manner.

• Using ChatGPT to speed up the writing of drafts for several outlets (reports, abstracts, emails, conference proposals, press releases, recommendation letters, etc.) ChatGPT can produce elaborated writing that must be edited to remove any possible inconsistencies or inaccuracies (December 7th, 2022 Social Science Space story )
• Using ChatGPT in the process of brainstorming ideas for curriculum design, lesson planning, and learning activities. The tool can provide some novel ideas or remind educators of some instructional techniques and strategies that they had heard about in the past (January 23rd, 2023, Article by Dr. David Wiley ).
• Using ChatGPT to provide students tutoring/scaffolds. The tool can act like a virtual tutor who does not simply give the answer to the student but rather scaffold them to reach the correct answers by themselves. (Sal Khan, founder/CEO of Khan Academy, Spring 2023 TED Talk )
• Teaching with ChatGPT to train students on using AI tools and models, provide opportunities to exercise critical thinking skills, and improve their technological literacy (January 12th New York Times story ).

Concluding Thoughts

There are major concerns about ChatGPT and the larger AI-powered Language Learning Models (LLMs) phenomenon. These concerns are legitimate and are opposed by notable ideas about the positive implications of AI-powered LLMs in higher education classrooms. As we aspire to make evidence-based educational and learning design decisions, one should carefully review the research that has been done on AI in relation to higher education up to this point and engage with the gaps as opportunities to expand knowledge and find new opportunities and risks.

Our University’s newly formed advisory committee on the applications of generative AI is a good example of how higher education institutions ought to recommend the use, evaluation, and development of emergent AI tools and services. Additionally, discussions about generative AI and its implications on education happening in public venues are necessary to strengthen the public-facing mission of the University, where input from educators, students, and members of the community is welcome.

Related Articles

Generative AI for Course Design: Crafting Learning Objectives

Generative AI for Course Design: The Basics