guidelines to critique a research article

SPH Writing Support Services

• Appointment System
• ESL Conversation Group
• Mini-Courses
• Thesis/Dissertation Writing Group
• Career Writing
• Citing Sources
• Critiquing Research Articles
• Project Planning for the Beginner This link opens in a new window
• Grant Writing
• Publishing in the Sciences
• Systematic Review Overview
• Systematic Review Resources This link opens in a new window
• Writing Across Borders / Writing Across the Curriculum
• Conducting an article critique for a quantitative research study: Perspectives for doctoral students and other novice readers (Vance et al.)
• Critique Process (Boswell & Cannon)
• The experience of critiquing published research: Learning from the student and researcher perspective (Knowles & Gray)
• A guide to critiquing a research paper. Methodological appraisal of a paper on nurses in abortion care (Lipp & Fothergill)
• Step-by-step guide to critiquing research. Part 1: Quantitative research (Coughlan et al.)
• Step-by-step guide to critiquing research. Part 2: Qualitative research (Coughlan et al.)

Guidelines:

• Critiquing Research Articles (Flinders University)
• Framework for How to Read and Critique a Research Study (American Nurses Association)
• How to Critique a Journal Article (UIS)
• How to Critique a Research Paper (University of Michigan)
• How to Write an Article Critique
• Research Article Critique Form
• Writing a Critique or Review of a Research Article (University of Calgary)

Presentations:

• The Critique Process: Reviewing and Critiquing Research
• Writing a Critique
• << Previous: Citing Sources
• Next: Project Planning for the Beginner >>
• Last Updated: Feb 12, 2024 1:11 PM
• URL: https://libguides.sph.uth.tmc.edu/writing_support_services

• All eBooks & Audiobooks
• Academic eBook Collection
• Home Grown eBook Collection
• Off-Campus Access
• Literature Resource Center
• Opposing Viewpoints
• ProQuest Central
• Course Guides
• Citing Sources
• Library Research
• Websites by Topic
• Book-a-Librarian
• Research Tutorials
• Use the Catalog
• Use Databases
• Use Films on Demand
• Use Home Grown eBooks
• Use NC LIVE
• Evaluating Sources
• Primary vs. Secondary
• Scholarly vs. Popular
• Make an Appointment
• Writing Tools
• Annotated Bibliographies
• Summaries, Reviews & Critiques
• Writing Center

Service Alert

Article Summaries, Reviews & Critiques

• Writing an article SUMMARY
• Writing an article REVIEW

Writing an article CRITIQUE

• Citing Sources This link opens in a new window
• About RCC Library

Text: 336-308-8801

Email: [email protected]

Call: 336-633-0204

Schedule: Book-a-Librarian

Like us on Facebook

Links on this guide may go to external web sites not connected with Randolph Community College. Their inclusion is not an endorsement by Randolph Community College and the College is not responsible for the accuracy of their content or the security of their site.

A critique asks you to evaluate an article and the author’s argument. You will need to look critically at what the author is claiming, evaluate the research methods, and look for possible problems with, or applications of, the researcher’s claims.

Introduction

Give an overview of the author’s main points and how the author supports those points. Explain what the author found and describe the process they used to arrive at this conclusion.

Body Paragraphs

Interpret the information from the article:

• Does the author review previous studies? Is current and relevant research used?
• What type of research was used – empirical studies, anecdotal material, or personal observations?
• Was the sample too small to generalize from?
• Was the participant group lacking in diversity (race, gender, age, education, socioeconomic status, etc.)
• For instance, volunteers gathered at a health food store might have different attitudes about nutrition than the population at large.
• How useful does this work seem to you? How does the author suggest the findings could be applied and how do you believe they could be applied?
• How could the study have been improved in your opinion?
• Does the author appear to have any biases (related to gender, race, class, or politics)?
• Is the writing clear and easy to follow? Does the author’s tone add to or detract from the article?
• How useful are the visuals (such as tables, charts, maps, photographs) included, if any? How do they help to illustrate the argument? Are they confusing or hard to read?
• What further research might be conducted on this subject?

Try to synthesize the pieces of your critique to emphasize your own main points about the author’s work, relating the researcher’s work to your own knowledge or to topics being discussed in your course.

From the Center for Academic Excellence (opens in a new window), University of Saint Joseph Connecticut

Additional Resources

All links open in a new window.

Writing an Article Critique (from The University of Arizona Global Campus Writing Center)

How to Critique an Article (from Essaypro.com)

How to Write an Article Critique (from EliteEditing.com.au)

• << Previous: Writing an article REVIEW
• Next: Citing Sources >>
• Last Updated: Mar 15, 2024 9:32 AM
• URL: https://libguides.randolph.edu/summaries

A guide for critique of research articles

Following is the list of criteria to evaluate (critique) a research article. Please note that you should first summarize the paper and then evaluate different parts of it.

Most of the evaluation section should be devoted to evaluation of internal validity of the conclusions. Please add at the end a section entitled ''changes in the design/procedures if I want to replicate this study." Attach a copy of the original article to your paper.

Click here to see a an example (this is how you start) of a research critique.

Click here to see the original article.

The following list is a guide for you to organize your evaluation. It is recommended to organize your evaluation in this order. This is a long list of questions. You don’t have to address all questions. However, you should address highlighted questions . Some questions may not be relevant to your article.

Introduction

1.     Is there a statement of the problem?

2.     Is the problem “researchable”? That is, can it be investigated through the collection and analysis of data?

3.     Is background information on the problem presented?

4.     Is the educational significance of the problem discussed?

5.     Does the problem statement indicate the variables of interest and the specific relationship between those variables which are investigated? When necessary, are variables directly or operationally defined?

Review of Related Literature

1.     Is the review comprehensive?

2.     Are all cited references relevant to the problem under investigation?

3.     Are most of the sources primary, i.e., are there only a few or no secondary sources?

4.     Have the references been critically analyzed and the results of various studies compared and contrasted, i.e., is the review more than a series of abstracts or annotations?

5.     Does the review conclude with a brief summary of the literature and its implications for the problem investigated?

6.     Do the implications discussed form an empirical or theoretical rationale for the hypotheses which follow?

1.     Are specific questions to be answered listed or specific hypotheses to be tested stated?

2.     Does each hypothesis state an expected relationship or difference?

3.     If necessary, are variables directly or operationally defined?

4.     Is each hypothesis testable?

Method          Subjects

1.     Are the size and major characteristics of the population studied described?

2.     If a sample was selected, is the method of selecting the sample clearly described?

3.      Is the method of sample selection described one that is likely to result in a representative, unbiased sample?

4.     Did the researcher avoid the use of volunteers?

5.     Are the size and major characteristics of the sample described?

6.     Does the sample size meet the suggested guideline for minimum sample size appropriate for the method of research represented?      

Instruments

1.     Is the rationale given for the selection of the instruments (or measurements) used?

2.     Is each instrument described in terms of purpose and content?

3.     Are the instruments appropriate for measuring the intended variables?

4.     Is evidence presented that indicates that each instrument is appropriate for the sample under study?

5.     Is instrument validity discussed and coefficients given if appropriate?

6.     Is reliability discussed in terms of type and size of reliability coefficients?

7.     If appropriate, are subtest reliabilities given?

8.     If an instrument was developed specifically for the study, are the procedures involved in its development and validation described?

9.     If an instrument was developed specifically for the study, are administration, scoring or tabulating, and interpretation procedures fully described?

Design and Procedure

1.     Is the design appropriate for answering the questions or testing the hypotheses of the   study?

2.     Are the procedures described in sufficient detail to permit them to be replicated by another researcher?

3.     If a pilot study was conducted, are its execution and results described as well as its impact on the subsequent study?

4.     Are the control procedures described?

5.     Did the researcher discuss or account for any potentially confounding variables that he or she was unable to control for?

1.     Are appropriate descriptive or inferential statistics presented?

2.     Was the probability level, α, at which the results of the tests of significance were evaluated,

       specified in advance of the data analyses?

3.     If parametric tests were used, is there evidence that the researcher avoided violating the

       required assumptions for parametric tests?

4.     Are the tests of significance described appropriate, given the hypotheses and design of the

       study?

5.     Was every hypothesis tested?

6.     Are the tests of significance interpreted using the appropriate degrees of freedom?

7.     Are the results clearly presented?

8.     Are the tables and figures (if any) well organized and easy to understand?

9.     Are the data in each table and figure described in the text?

Discussion (Conclusions and Recommendation)

1.     Is each result discussed in terms of the original hypothesis to which it relates?

2.     Is each result discussed in terms of its agreement or disagreement with previous results

        obtained by other researchers in other studies?

3.     Are generalizations consistent with the results?

4.     Are the possible effects of uncontrolled variables on the results discussed?

5.     Are theoretical and practical implications of the findings discussed?

6.     Are recommendations for future action made?

7.     Are the suggestions for future action based on practical significance or on statistical

       significance only, i.e., has the author avoided confusing practical and statistical

       significance?

8.     Are recommendations for future research made?

Additional general questions to be answered in your critique.

1. What is (are) the research question(s) (or hypothesis)?

2. Describe the sample used in this study.

3. Describe the reliability and validity of all the instruments used.

4. What type of research is this?  Explain.

5. How was the data analyzed?

6. What is (are) the major finding(s)?

Reading and critiquing a research article

Nurses use research to answer questions about their practice, solve problems, improve the quality of patient care, generate new research questions, and shape health policy. Nurses who confront questions about practice and policy need strong, high-quality, evidence-based research. Research articles in peer-reviewed journals typically undergo a rigorous review process to ensure scholarly standards are met. Nonetheless, standards vary among reviewers and journals. This article presents a framework nurses can use to read and critique a research article.

When deciding to read an article, determine if it’s about a question you have an interest in or if it can be of use in your practice. You may want to have a research article available to read and critique as you consider the following questions.

Does the title accurately describe the article?

A good title will pique your interest but typically you will not know until you are done reading the article if the title is an accurate description. An informative title conveys the article’s key concepts, methods, and variables.

Is the abstract representative of the article?

The abstract provides a brief overview of the purpose of the study, research questions, methods, results, and conclusions. This helps you decide if it’s an article you want to read. Some people use the abstract to discuss a study and never read further. This is unwise because the abstract is just a preview of the article and may be misleading.

Does the introduction make the purpose of the article clear?

A good introduction provides the basis for the article. It includes a statement of the problem, a rationale for the study, and the research questions. When a hypothesis is being tested, it should be clearly stated and include the expected results.

Is a theoretical framework described?

When a theoretical framework is used, it should inform the study and provide a rationale. The concepts of the theoretical framework should relate to the topic and serve as a basis for interpreting the results. Some research doesn’t use a theoretical framework, such as health services research, which examines issues such as access to care, healthcare costs, and healthcare delivery. Clinical research such as comparing the effectiveness of two drugs won’t include a theoretical framework.

Is the literature review relevant to the study and comprehensive? Does it include recent research?

The literature review provides a context for the study. It establishes what is, and is not known about the research problem. Publication dates are important but there are caveats. Most literature reviews include articles published within the last 3 to 5 years. It can take more than a year for an article to be reviewed, revised, accepted, and published, causing some references to seem outdated.

Literature reviews may include older studies to demonstrate important changes in knowledge over time. In an area of study where little or no research has been conducted, there may be only a few relevant articles that are a decade or more old. In an emerging area of study there may be no published research, in which case related research should be referenced. If you are familiar with the area of research, review the references to determine if well-known and highly regarded studies are included.

Does the methods section explain how a research question was addressed?

The methods section provides enough information to allow the study to be replicated. Components of this section indicate if the design is appropriate to answer the research question(s).

• Did the researcher select the correct sample to answer the research questions and was the size sufficient to obtain valid results?
• If a data collection instrument was used, how was it created and validated?
• If any materials were used, such as written guides or equipment, were they described?
• How were data collected?
• Was reliability and validity accounted for?
• Were the procedures listed in a step-by-step manner?

Independent and dependent variables should be described and terms defined. For example, if patient falls in the hospital are considered the dependent variable, or outcome, what are the independent variables, or factors, being investigated that may influence the rate at which patient falls occur? In this example, independent variables might include nurse staffing, registered nurse composition (such as education and certification), and hospital Magnet &#174 status.

Is the analytical approach consistent with the study questions and research design?

The analytical approach relates to the study questions and research design. A quantitative study may use descriptive statistics to summarize the data and other tests, such as chi squares, t-tests, or regression analysis, to compare or evaluate the data. A qualitative study may use such approaches as coding, content analysis, or grounded theory analysis. A reader who is unfamiliar with the analytical approach may choose to rely on the expertise of the journal’s peer reviewers who assessed whether the analytical approach was correct.

Are the results presented clearly in the text and in tables and figures?

Results should be clearly summarized in the text, tables, and figures. Tables and figures are only a partial representation of the results and critical information may be only in the text. In a quantitative study, the significance of the statistical tests is important. The presentation of qualitative results should avoid interpretation, which is reserved for the discussion.

Are the limitations presented and their implications discussed?

It is essential that the limitations of the study be presented. These are the factors that explain why the results may need to be carefully interpreted, may only be generalized to certain situations, or may provide less robust results than anticipated. Examples of limitations include a low response rate to a survey, not being able to establish causality when a cross-sectional study design was used, and having key stakeholders refuse to be interviewed.

Does the discussion explain the results in relation to the theoretical framework, research questions, and significance of the study?

The discussion serves as an opportunity to explain the results in respect to the research questions and the theoretical framework. Authors use the discussion to interpret the results and explain the meaning and significance of the study. It’s also important to distinguish the study from others that preceded it and provide recommendations for future research.

Depending on the research, it may be equally important for the investigators to present the clinical and/or practical significance of the results. Relevant policy recommendations are also important. Evaluate if the recommendations are supported by the data or seem to be more of an opinion. A succinct conclusion typically completes the article.

Once you’re done reading the article, how do you decide if the research is something you want to use?

Determine the scientific merit of the study by evaluating the level and quality of the evidence. There are many scales to use, several of which can be found in the Research Toolkit on the American Nurses Association’s website http://www.nursingworld.org/research-toolkit.aspx . Consider what you learned and decide if the study is relevant to your practice or answered your question as well as whether you can implement the findings.

A new skill

A systematic approach to reading and critiquing a research article serves as a foundation for translating evidence into practice and policy. Every nurse can acquire this skill.

Louise Kaplan is director of the nursing program at Saint Martin’s University in Lacey, Washington. At the end of this article is a checklist for evaluating an article.

Selected references

Hudson-Barr D. How to read a research article. J Spec Pediatr Nurs . 2004;9(2):70-2.

King’s College D. Leonard Corgan Library. Reading a research article. http://www.lib.jmu.edu/ilworkshop08/materials/studyguide3.pdf . Accessed September 5, 2012.

Oliver D, Mahon SM. Reading a research article part I: Types of variables. Clin J Oncol Nurs . 2005;9(1):110-12.

Oliver D, Mahon SM. Reading a research article part II: Parametric and nonparametric statistics. Clin J Oncol Nurs . 2005;9(2):238-240.

Oliver D, Mahon SM. Reading a research article part III: The data collection instrument. Clin J Oncol Nurs . 2006;10(3):423-26.

Rumrill P, Fitzgerald S, Ware, M. Guidelines for evaluating research articles. Work . 2000;14(3):257-63.

15 Comments .

very helpful resource to critique any research article

I like it helped me a lot in my critical appraisal. thank you very much.

This article will help me with my understanding of how to read and critique a research article. This article was helpful in breaking down this information very basic to get a clear, concise understanding. Now I can take this information and go to the next level in my discussions

Great information and I will use this article for future reference.

This checklist and explanation for a literature review and/or reading and critiquing a research article was very helpful. As I only have 2 more classes to get my degree, I wish I knew this info 2 semesters ago! I will also pass this along to coworkers that will be going back to school in the near future.

Great article, I enjoyed the information. Thank You for this resource. Carolyn Martinez

Fantastic guide to the interpretation of clinical trials. Found this so helpful!

Great information and article. Thank you for the information.

well explained. its sometimes hard for P.G students to understand the concept but these guidelines are helpful to learn for novice.

This is great,am looking for guilgline on how to do research critique and this is just the solution.Thnks weldone

Unsure how to appropriately critique an article, thank you for your infomation

I am currently taking a Health Service Research course and was not sure how to sturcture my assignment. Thanks for posting this article!

very informative…very helpful to students doing research work.

Great timing; have just been asked to review and article and you provide the guide! Will share with colleagues.

I will be passing this article on to a friend who is taking a nursing research class. This article is a great reference for nursing students.

Comments are closed.

NurseLine Newsletter

• First Name *
• Last Name *
• Hidden Referrer

*By submitting your e-mail, you are opting in to receiving information from Healthcom Media and Affiliates. The details, including your email address/mobile number, may be used to keep you informed about future products and services.

Test Your Knowledge

Recent posts.

Nursing professional development at night

Engaging nurses in scholarly work

Mentorship matters

2023 nursing trends and salary survey results

NCLEX vs Next Generation

Patient care assistant training

Re-imagining nursing’s social contract with the public

Is there a nurse in the House? Or the Senate?

Prevent compassion fatigue through self-compassion

Implementing trauma-informed care 

Addressing transgender myths

Motivate Gen Z nursing students

Postural orthostatic tachycardia syndrome

Pulmonary hypertension: Consider the “zebra”

2021 CDC STI guidelines: A review of changes

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Turk J Urol
• v.39(Suppl 1); 2013 Sep

How to write a review article?

In the medical sciences, the importance of review articles is rising. When clinicians want to update their knowledge and generate guidelines about a topic, they frequently use reviews as a starting point. The value of a review is associated with what has been done, what has been found and how these findings are presented. Before asking ‘how,’ the question of ‘why’ is more important when starting to write a review. The main and fundamental purpose of writing a review is to create a readable synthesis of the best resources available in the literature for an important research question or a current area of research. Although the idea of writing a review is attractive, it is important to spend time identifying the important questions. Good review methods are critical because they provide an unbiased point of view for the reader regarding the current literature. There is a consensus that a review should be written in a systematic fashion, a notion that is usually followed. In a systematic review with a focused question, the research methods must be clearly described. A ‘methodological filter’ is the best method for identifying the best working style for a research question, and this method reduces the workload when surveying the literature. An essential part of the review process is differentiating good research from bad and leaning on the results of the better studies. The ideal way to synthesize studies is to perform a meta-analysis. In conclusion, when writing a review, it is best to clearly focus on fixed ideas, to use a procedural and critical approach to the literature and to express your findings in an attractive way.

The importance of review articles in health sciences is increasing day by day. Clinicians frequently benefit from review articles to update their knowledge in their field of specialization, and use these articles as a starting point for formulating guidelines. [ 1 , 2 ] The institutions which provide financial support for further investigations resort to these reviews to reveal the need for these researches. [ 3 ] As is the case with all other researches, the value of a review article is related to what is achieved, what is found, and the way of communicating this information. A few studies have evaluated the quality of review articles. Murlow evaluated 50 review articles published in 1985, and 1986, and revealed that none of them had complied with clear-cut scientific criteria. [ 4 ] In 1996 an international group that analyzed articles, demonstrated the aspects of review articles, and meta-analyses that had not complied with scientific criteria, and elaborated QUOROM (QUality Of Reporting Of Meta-analyses) statement which focused on meta-analyses of randomized controlled studies. [ 5 ] Later on this guideline was updated, and named as PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). [ 6 ]

Review articles are divided into 2 categories as narrative, and systematic reviews. Narrative reviews are written in an easily readable format, and allow consideration of the subject matter within a large spectrum. However in a systematic review, a very detailed, and comprehensive literature surveying is performed on the selected topic. [ 7 , 8 ] Since it is a result of a more detailed literature surveying with relatively lesser involvement of author’s bias, systematic reviews are considered as gold standard articles. Systematic reviews can be diivded into qualitative, and quantitative reviews. In both of them detailed literature surveying is performed. However in quantitative reviews, study data are collected, and statistically evaluated (ie. meta-analysis). [ 8 ]

Before inquring for the method of preparation of a review article, it is more logical to investigate the motivation behind writing the review article in question. The fundamental rationale of writing a review article is to make a readable synthesis of the best literature sources on an important research inquiry or a topic. This simple definition of a review article contains the following key elements:

• The question(s) to be dealt with
• Methods used to find out, and select the best quality researches so as to respond to these questions.
• To synthetize available, but quite different researches

For the specification of important questions to be answered, number of literature references to be consulted should be more or less determined. Discussions should be conducted with colleagues in the same area of interest, and time should be reserved for the solution of the problem(s). Though starting to write the review article promptly seems to be very alluring, the time you spend for the determination of important issues won’t be a waste of time. [ 9 ]

The PRISMA statement [ 6 ] elaborated to write a well-designed review articles contains a 27-item checklist ( Table 1 ). It will be reasonable to fulfill the requirements of these items during preparation of a review article or a meta-analysis. Thus preparation of a comprehensible article with a high-quality scientific content can be feasible.

PRISMA statement: A 27-item checklist

Contents and format

Important differences exist between systematic, and non-systematic reviews which especially arise from methodologies used in the description of the literature sources. A non-systematic review means use of articles collected for years with the recommendations of your colleagues, while systematic review is based on struggles to search for, and find the best possible researches which will respond to the questions predetermined at the start of the review.

Though a consensus has been reached about the systematic design of the review articles, studies revealed that most of them had not been written in a systematic format. McAlister et al. analyzed review articles in 6 medical journals, and disclosed that in less than one fourth of the review articles, methods of description, evaluation or synthesis of evidence had been provided, one third of them had focused on a clinical topic, and only half of them had provided quantitative data about the extend of the potential benefits. [ 10 ]

Use of proper methodologies in review articles is important in that readers assume an objective attitude towards updated information. We can confront two problems while we are using data from researches in order to answer certain questions. Firstly, we can be prejudiced during selection of research articles or these articles might be biased. To minimize this risk, methodologies used in our reviews should allow us to define, and use researches with minimal degree of bias. The second problem is that, most of the researches have been performed with small sample sizes. In statistical methods in meta-analyses, available researches are combined to increase the statistical power of the study. The problematic aspect of a non-systematic review is that our tendency to give biased responses to the questions, in other words we apt to select the studies with known or favourite results, rather than the best quality investigations among them.

As is the case with many research articles, general format of a systematic review on a single subject includes sections of Introduction, Methods, Results, and Discussion ( Table 2 ).

Structure of a systematic review

Preparation of the review article

Steps, and targets of constructing a good review article are listed in Table 3 . To write a good review article the items in Table 3 should be implemented step by step. [ 11 – 13 ]

Steps of a systematic review

The research question

It might be helpful to divide the research question into components. The most prevalently used format for questions related to the treatment is PICO (P - Patient, Problem or Population; I-Intervention; C-appropriate Comparisons, and O-Outcome measures) procedure. For example In female patients (P) with stress urinary incontinence, comparisons (C) between transobturator, and retropubic midurethral tension-free band surgery (I) as for patients’ satisfaction (O).

Finding Studies

In a systematic review on a focused question, methods of investigation used should be clearly specified.

Ideally, research methods, investigated databases, and key words should be described in the final report. Different databases are used dependent on the topic analyzed. In most of the clinical topics, Medline should be surveyed. However searching through Embase and CINAHL can be also appropriate.

While determining appropriate terms for surveying, PICO elements of the issue to be sought may guide the process. Since in general we are interested in more than one outcome, P, and I can be key elements. In this case we should think about synonyms of P, and I elements, and combine them with a conjunction AND.

One method which might alleviate the workload of surveying process is “methodological filter” which aims to find the best investigation method for each research question. A good example of this method can be found in PubMed interface of Medline. The Clinical Queries tool offers empirically developed filters for five different inquiries as guidelines for etiology, diagnosis, treatment, prognosis or clinical prediction.

Evaluation of the Quality of the Study

As an indispensable component of the review process is to discriminate good, and bad quality researches from each other, and the outcomes should be based on better qualified researches, as far as possible. To achieve this goal you should know the best possible evidence for each type of question The first component of the quality is its general planning/design of the study. General planning/design of a cohort study, a case series or normal study demonstrates variations.

A hierarchy of evidence for different research questions is presented in Table 4 . However this hierarchy is only a first step. After you find good quality research articles, you won’t need to read all the rest of other articles which saves you tons of time. [ 14 ]

Determination of levels of evidence based on the type of the research question

Formulating a Synthesis

Rarely all researches arrive at the same conclusion. In this case a solution should be found. However it is risky to make a decision based on the votes of absolute majority. Indeed, a well-performed large scale study, and a weakly designed one are weighed on the same scale. Therefore, ideally a meta-analysis should be performed to solve apparent differences. Ideally, first of all, one should be focused on the largest, and higher quality study, then other studies should be compared with this basic study.

Conclusions

In conclusion, during writing process of a review article, the procedures to be achieved can be indicated as follows: 1) Get rid of fixed ideas, and obsessions from your head, and view the subject from a large perspective. 2) Research articles in the literature should be approached with a methodological, and critical attitude and 3) finally data should be explained in an attractive way.

Guidelines for critique of a research report

Affiliation.

• 1 Department of Life Span Process, College of Nursing, Ohio State University, Columbus.
• PMID: 1629010

Before findings reported in a research article are used to change clinical practice, the research must be evaluated for evidence of credibility, integrity, and potential for replication. This paper provides guidelines for use in evaluating deductive, empirical research. The standards presented in this paper are derived from the tenets of the scientific method, measurement theory, statistical principles, and research ethics. The guidelines may be used to evaluate each section of a research report, from the title to the interpretation of findings.

• Nursing Research*
• Research Design

Developing a Framework for Self-regulatory Governance in Healthcare AI Research: Insights from South Korea

• Perspective
• Open access
• Published: 25 March 2024

Cite this article

You have full access to this open access article

• Junhewk Kim   ORCID: orcid.org/0000-0002-9109-270X 1 ,
• So Yoon Kim   ORCID: orcid.org/0000-0001-7015-357X 2 ,
• Eun-Ae Kim   ORCID: orcid.org/0000-0002-6989-559X 3 ,
• Jin-Ah Sim   ORCID: orcid.org/0000-0002-3494-3002 4 ,
• Yuri Lee   ORCID: orcid.org/0000-0003-0584-650X 5 &
• Hannah Kim   ORCID: orcid.org/0000-0003-2938-9745 2  

This paper elucidates and rationalizes the ethical governance system for healthcare AI research, as outlined in the ‘Research Ethics Guidelines for AI Researchers in Healthcare’ published by the South Korean government in August 2023. In developing the guidelines, a four-phase clinical trial process was expanded to six stages for healthcare AI research: preliminary ethics review (stage 1); creating datasets (stage 2); model development (stage 3); training, validation, and evaluation (stage 4); application (stage 5); and post-deployment monitoring (stage 6). Researchers identified similarities between clinical trials and healthcare AI research, particularly in research subjects, management and regulations, and application of research results. In the step-by-step articulation of ethical requirements, this similarity benefits from a reliable and flexible use of existing research ethics governance resources, research management, and regulatory functions. In contrast to clinical trials, this procedural approach to healthcare AI research governance effectively highlights the distinct characteristics of healthcare AI research in research and development process, evaluation of results, and modifiability of findings. The model exhibits limitations, primarily in its reliance on self-regulation and lack of clear delineation of responsibilities. While formulated through multidisciplinary deliberations, its application in the research field remains untested. To overcome the limitations, the researchers’ ongoing efforts for educating AI researchers and public and the revision of the guidelines are expected to contribute to establish an ethical research governance framework for healthcare AI research in the South Korean context in the future.

Avoid common mistakes on your manuscript.

Introduction

The rapid progress of machine learning and artificial intelligence (AI) poses new and unprecedented challenges to the entire healthcare sector. Particularly, as a critical extension of the foundational discussions on the technology adoption in healthcare (Rajpurkjar et al. 2022 ), the focus now shifts towards the practical governance and regulation of AI development and its application in healthcare landscape. South Korea has swiftly embraced biomedical technologies, showcasing a clear inclination in integrating AI in healthcare. The ‘2022 Medical Device License Report’ from the Ministry of Food and Drug Safety (MFDS) of the Republic of Korea unveils that a total of 149 AI-based medical devices obtained approval and certification in the country, with 10 receiving approval and 38 attaining certification in 2022 (MFDS 2023 ).

Corresponding with this trend, the Korean National Institutes of Health (KNIH) published the ‘Research Ethics Guidelines for AI Researchers in Healthcare’ in August 2023, marking an initial effort to offer an actionable guidance to healthcare AI researchers in the country (KNIH 2023 ). The guidelines aim to establish ethical standards for all stages of healthcare AI development by presenting ethical principles and detailed values. The researchers mainly participated in developing the guidelines using robust research methodologies, such as literature reviews, interdisciplinary consultations, and a public hearing as well as providing empirical research evidence from surveys for the lay public and experts. Consequently, the guidelines present six principles with corresponding codes and explanations. The principles, stemmed from the World Health Organization (WHO) report ‘Ethics and governance of artificial intelligence for health’, are tailored for the national context, providing a framework for researchers to evaluate their research practices. Importantly, it is noted that while bioscientists are well-versed in the ethical procedures and legal regulations related to human subjects research, those in computer science and data science engaged in healthcare AI research may lack familiarity with these standards (Metcalf and Crawford 2016 ; Throne 2022 ). Consequently, these guidelines are designed to assist support healthcare AI researchers in conducting ethical research by presenting providing part I principles to consider in relation to research, part II corresponding relevant research codes, regulations and related ethical cases, and part III an expanded framework, aligning with that applies for the existing governance framework for phase I–IV clinical research, tailored for to the context of healthcare AI research.

The purpose of this paper is to outline and provide rationales for the ethical governance system introduced in the part III of the guidelines. At present, we are in the process of translating the guidelines for an official English version. Amidst this ongoing endeavour, this paper preliminarily introduces the final section of the guidelines, which is under linguistic review. Subsequently, we describe the governance framework, comprising six steps, accompanied by ethical and institutional explanations for each stage. In conclusion, this paper presents a healthcare AI research governance system, expanding upon the existing human subjects research. It advocates for the establishment of a robust, secure, and sustainable research governance structure by adapting the clinical research system prevalent in countries where such approaches are already established to the domain of healthcare AI research governance.

Procedural Considerations for Conducting Healthcare AI Research

Aforementioned, part I of the guidelines provide background, developing process and methodologies, aims, scope, and key terms. Next, part II reviews the existing legal frameworks related to safety and effectiveness, liability for errors and negligence, privacy laws for patient data protection, and legal frameworks responding to bias and discrimination. Based on the legal background, part III introduces a six principle-based framework and explanations with specific ethical cases, aligning with the procedural considerations when researchers conduct healthcare AI research.

Particularly, part III of the guidelines is grounded in six ethical principles: (a) respect for and protection of human autonomy; (b) promotion of human well-being, safety, and the public interest; (c) ensuring transparency, explainability, and reliability; (d) upholding accountability and legal obligations; (e) promoting inclusivity and equity; and (f) fostering responsiveness and sustainability (Kim et al. 2023 ). While these principles align with those of the WHO, specific codes and applications have been tailored to suit the national context. The healthcare AI research governance framework presented herein also follows this approach, incorporating relevant principles to be considered at each stage.

The guidelines restructured the principles by the steps of the research process as a form of a checklist. This checklist provides a baseline for all stakeholders the field to voluntarily identify and assess the ethical considerations pertinent to practical research and development (Table  1 ).

Healthcare AI research and development begins with the establishment of a robust ethical framework, grounded in the aforementioned six ethical principles. A multidisciplinary team collaborated to establish the ethical considerations for research and development and delineate the requisite compliance measures. The AI development process comprises distinct stages: data collection, algorithm development, model training integration, and evaluation. Each steps follows a structured ethical framework, integrating the principles, thereby ensuring the ethical integrity of healthcare AI research and development. Periodic evaluations are conducted to assess ethical compliance and identify areas for improvement. Furthermore, continuous feedback is sought following the application of the developed model in real-world environment.

For research institutions, the guidelines play a pivotal role in ensuring ethical standards of healthcare AI research and development. The research institutions can utilize the guidelines to evaluate the design procedure, algorithm development, and application of AI technologies in their own research endeavours. This assessment entails evaluating the alignment of the guidelines with domestic laws, international norms, and societal dialogues. Additionally, it is advisable for review committees and institutions that oversights healthcare AI research and development to implement reasonable and responsible regulations to manage research activities, educating and informing stakeholders about these regulations, and maintaining open communication for ongoing revisions and amendments as required.

Furthermore, through such feedback and societal discussion, the developers of this guidelines strive for continuous refinement, aiming to foster a research environment that esteems ethical principles and values.

Stage 1. Preliminary Ethics Review

Prior to the commencement of healthcare AI research and development, it is imperative to establish a clear ethical framework guided by specific guidelines. This preliminary stage is the responsibility of the organization, tasked with laying the foundational groundwork. They should actively seek advice through public participation action from a diverse array of stakeholders, including patients, the public, and expert groups such as medical ethicists and legal scholars, to ensure a well-rounded perspective through public participation action. Additionally, it is essential to establish and consider ethical guidelines that are particularly relevant to the research and development process, setting a strong foundation for responsible and ethical AI innovation in healthcare.

Related questions:

Does the plan include sensitive objectives? Is the objective to develop a medical device or other health and public health objectives? (Specify clinical diagnosis-treatment decision, patient decision support, prevention, behavioural intervention, public health, and if others, additional descriptions should be included in the protocol.)

Is it human subject research or research utilizing datasets? (check bioethics exemptions and compliance requirements.) If human subjects research, does the plan include interventions or interactions?

Does the plan address potential or manifest harms? (Provide a risk-benefit analysis.)

Is there evidence or potential for sample bias in the plan?

Stage 2. Creating Datasets

In the process of collecting and processing data for healthcare AI model development, several key considerations must be addressed. Initially, it is essential to evaluate the collectability, availability, and intended use of the data. Depending on the potential risk for privacy infringement, appropriate measures such as anonymization or pseudonymization should be employed for the dataset. A detailed data collection plan is crucial to outline the methods and objectives clearly. Additionally, conducting ongoing quality control is imperative to minimize data bias and ensure the diversity and representativeness of the datasets, which are fundamental for the development of fair and effective healthcare AI systems.

Is the data collection plan comprehensive? (identification and consultation with data subjects or maintaining organizations, data types and details, collection techniques, frequency selection, inclusion and appropriateness of purposes of use)

Are anonymization measures considered? (detailed technical and administrative/physical measures; if not anonymized, justification and additional measures required)

Is the dataset size aligned with the learning task and model complexity?

Is the data quality recognized as high?

Are the data appropriately visualized and exploratory analyses conducted?

Is the raw data collected according to approved clinical standards and protocols, utilizing valid and reliable techniques?

Are regular and continuous data quality control measures implemented?

Stage 3. Model Development

Configuring algorithms to align with research objectives and applying preliminary data to assess appropriateness is a critical phase in AI development. Developers should build the model using decision-making algorithms aimed at achieving specific, predefined goals. To ensure transparency, a concise description of the development plan should be publicized, detailing the steps and intentions behind the model’s construction. Standardizing the data before training the model is essential to ensure consistency and accuracy. Additionally, it is crucial to specify any methodological considerations that might reveal bias within the dataset, thereby allowing for adjustments and improvements to maintain integrity and fairness in the model's outcomes.

Related question (considerations in Stage 1 should be considered in conjunction with those below)

Does the plan provide an adequate accounting of human subjects and data subjects?

Are the methods of split cross-validation of datasets and datasets utilized in the plan appropriate? (correcting erroneous data, resolving inconsistencies in data, deleting unnecessary data, ensuring quality assurance and accuracy of data)

Are potential issues with privacy addressed? (review for possible data breach)

Does the plan assess the sources or likelihood of sampling/evaluation/algorithmic bias? (considering resampling, algorithmic fairness, etc.)

Stage 4. Training, Validation, and Evaluation

The phase of training and validating algorithms using the collected data, followed by an evaluation of their applicability for research purposes, is crucial for crafting robust AI systems. Training AI models meticulously is fundamental to boost their reliability and accuracy. It is also critical to ensure that the AI models undergo thorough internal validation through appropriate procedures to confirm its effectiveness and safety in practical applications. Moreover, implementing measures to assess clinical reliability is necessary for healthcare AI development. This includes evaluating the AI’s accuracy, its relevance to clinical applications, the fairness of its decision-making processes, and the level of trust or acceptance these systems receive from both patients and healthcare professionals.

Does the model use a transparent methodology for AI data mining and project implementation? (e.g., CRISP-DM, Footnote 1 KDD, Footnote 2 SEMMA, Footnote 3 CPMAI Footnote 4 )

What is the model’s purpose? (specify predictive models, text mining, automation, record abstraction, biometrics, and if others, additional descriptions should be in the protocol)

What kind of technology is utilized? (specify machine learning, deep learning, natural language processing, unsupervised learning, reinforcement learning, and if others, additional descriptions should be included in the protocol.)

Can any unexpected results be analysed or tracked?

Stage 5. Application

Ensuring compliance with ethical frameworks and legal regulations is paramount when governing AI models in the real-world application. AI models functioning as medical devices, tasked with analysing data for disease diagnosis, management, and prediction, must comply with approval and review protocols established by relevant regulatory bodies. Those covered by health insurance require safety, effectiveness, and economic evaluations by designated authorities. Implementing an external validation process that involves public participation can further reinforce the model’s integrity and social acceptance.

Furthermore, it becomes crucial that clinical AI algorithms to prioritize user-friendliness, requiring minimal training to lessen cognitive load and streamline decision-making. Supervising and maintaining the models involve assessing their ethical integrity and making continuous improvements as necessary. Clearly designate a specific individual or entity responsible for the ethical management of the model.

Is there a match between the dataset and the population setting for model application?

Are the results interpretable?

Have they been assessed for major biases? (e.g., gender, race)

Has the model been externally validated using datasets from other settings?

Has the model been empirically evaluated for validity, clinical utility, and cost-effectiveness?

Stage 6. Post-deployment Monitoring

Continuing engagement with model users and refining the model based on their feedback is essential in this stage. It involves regularly reviewing the model’s performance in real-world applications, aligning with the self-constructed ethical framework previously established. Maintaining open communication and collaboration with all stakeholders, including AI providers, users, patients, the public, and government agencies, is crucial for ongoing development and alignment with user needs and ethical standards. Furthermore, ensuring that the models can be seamlessly integrated into existing production environments is vital for effective decision-making based on real data. This stage emphasizes the importance of adaptability and responsiveness to the evolving landscape of AI applications and societal impacts.

Do you regularly monitor the product whether the entire data process is correctly aligned or when the entire process is performed automatically without the need for human intervention?

Does the user (healthcare provider), user organization (healthcare organization) regularly disclose usage results, both positive and negative?

Are there communication and recovery protocols established for model application errors?

Are there improvements needed in the relevant ethical framework and guidelines?

A Step-by-Step Explanation of Healthcare AI Research Governance Framework

The healthcare AI research governance framework delineated above adapts and extends the phase I–IV process for human clinical research to healthcare AI research. This adaptation allows guideline developers to manage and regulate research more reliably by extending existing research governance procedures, thus reducing the need for designing new schema for healthcare AI research ethics. This approach reduces training efforts and provides a foundation for researchers to quickly comprehend and apply the governance framework. Additionally, many of the administrative resources already established for human subjects research can be leveraged for healthcare AI research.

However, it is imperative to analyse the commonalities and divergences between clinical trials and healthcare AI research. This paper presents the similarities in terms of (a) research subjects, (b) areas of research management and regulation, and (c) application of research results. On the other hand, there are differences between clinical trials and healthcare AI research, including (a) the research and development process, (b) evaluation of research results, and (c) the modifiability of research results.

Firstly, human subjects, biospecimens, or populations in clinical trials share qualitative similarity with health data, their constructs, or databases utilized in healthcare AI research. For instance, biospecimens are recognized for their uniqueness—characteristics derived from the individuals they originate from—and then, health data collected from human subjects possess the same ontological nature as derivatives of individuals. They inherently refer to persons and are intricately connected to them (Cha and Kim 2022 ). Health datasets encapsulate various biological, behavioural, and socioeconomic records of a specific data subject, directly linked with the human body. The linkage of whole genome sequencing (WGS) data to personal identity intertwines the human body with the data presenting (Li et al. 2014 ). In population studies, the population database reflects the target population group, and eventually, they should become ontologically and practically identical.

Secondly, both clinical trials and healthcare AI research aim to derive results that benefit humans—whether it is treatments, new drugs, medical technologies, and biomaterials in clinical trials, or algorithms and applications in healthcare AI research. Just as clinical research with human subjects has established protocols to ensure respect and protection of individuals involved and affected by research process and its outcome (National Commission for the Protection of Human Subjects of Biomedical & Behavioral Research 1978 ), healthcare AI research also confronts to address ethical considerations arising from both the research process and the utilization of its outcomes. The considerations encompass aspects ranging from the respecting and protection of individuals to issue of accountability and sustainability. Similar to the human subjects research oversight by Institutional Review Boards (IRBs), which review and monitor all biomedical research, healthcare AI research necessitates a robust review and monitoring process. This process is crucial even when certain research activities might be exempt from regulatory requirements, acknowledging the unique challenges and potential risks associated with AI. A tailored oversight mechanism for healthcare AI is imperative that all research involving human subjects—or their data—is conducted responsibly and ethically. As human clinical trials aim to apply developed treatments and new drugs to humans by assessing efficacy and safety, healthcare AI research endeavours to apply developed algorithms and applications to humans to demonstrate effectiveness.

Recognizing the identified similarities, it could be argued that the governance framework established for human clinical research can be directly applied to healthcare AI research. However, significant differences between human clinical research and healthcare AI research necessitate a tailored approach.

Primarily, a distinction lies in the development process between human clinical research and healthcare AI research. Human clinical research focuses on developing of treatments or new drugs, validated through assessments of safety and effectiveness and comparative benefit analyses. Upon affirming these steps, a treatment or drug is considered developed, thereafter maintained through post-marketing/application monitoring or management. Conversely, healthcare AI research entails an iterative process of development, refinement, and validation of algorithms or applications, inherently characterized by their modifiability (Higgins and Madai 2020 ). This research paradigm encompasses a series of stages from data collection to algorithm application and continual revision through feedback loops. Throughout the progress, algorithms are expected to continuously learn, revise, and evolve (Pianykh et al. 2020 ). Therefore, a governance approach tailored to this process, spanning from data collection and algorithm development to model training integration, and evaluation becomes essential.

The primary difference consequently leads to variations in how research outcomes are evaluated and modified. Clinical trials typically employ statistical validation methods like randomized controlled trials (RCTs) or equivalent methodologies to confirm effectiveness. In contrast, healthcare AI research assesses performance using metrics such as the area under the receiver operating characteristic (ROC) curve (AUC) derived from collected data (Wu et al. 2021 ), which involves trade-offs between false positives and false negatives. In addition, drugs and medical devices approved through clinical trials are subject to re-evaluation if modifications are made. However, in healthcare AI research, accepting modifications poses a challenge due to its continuous learning nature, disrupting the notion of a consistent “product-based view” (Gerke et al. 2020 ). Therefore, it is practical for governing healthcare AI research governance to consider adopting elements maintainable from the human subjects clinical research governance system while modifying them to suit the development and application dynamics of healthcare AI.

Six-Stage Process for Healthcare AI Research

Given these considerations, this guidance extends the traditional four-phase clinical research process (phase I: safety; phase II: efficacy and side-effects; phase III: large trials; phase IV: post-market surveillance) by introducing a six-stage process for healthcare AI research. The introduction of <Stage 1: preliminary ethics review > and < Stage 2: creating datasets > reflects the unique nature of healthcare AI research and emphasizes the necessity for comprehensive and sustainable research guidelines from data collection stage onwards. < Stage 3: model development > , < Stage 4: training, validation, and evaluation > , < Stage 5: application > , and < Stage 6: post-deployment monitoring > align with the concepts of phases I–IV of clinical research but are specifically tailored to address the characterized process of developing and applying healthcare AI algorithms.

Stage 1 necessitates researchers and developers to establish an ethical framework tailored to their research objectives. This endeavour enables the research organizations and their members to review and establish their own ethical frameworks and establish and operate a framework that is appropriate for their research purposes. Given the diverse nature of healthcare AI, the selection and explicit delineation of an appropriate ethical framework are crucial. The first stage supports engagement of a diverse array of experts and the public, including ethicists, legal scholars, patients, and laypersons to take an interest in the AI research process as necessary. Their collective input serves to establish guiding principles and rules crucial for the ethical conduct of research. This proactive approach aims to promote self-regulated ethical practices among researchers, distinct from mere compliance with legal regulations. Notably, the established ethical framework in stage 1 should be consistently referenced in most subsequent documentation.

Stage 2 specifies plans for data collection and processing, mandating the creation of suitable datasets by designated data creator or “data curators” responsible for assembling and maintaining datasets (Leonelli 2016 ). The data collection and processing activities of researchers undergo to review by the Data Review Boards (DRBs). This board, established to oversight the ethical conduct of data-related procedures, evaluated data collection plan, anonymization methods, dataset size, quality, and management. The DRB operates within the research institution or as an independent body. Proposed by the Ministry of Health and Welfare of South Korea in the “Guidelines for Utilization of Healthcare Data,” the DRBs function as a committee of five or more individuals. Its responsibilities include assessing the suitability of processing pseudonymized information within an institution, reviewing the adequacy of pseudonymization, and managing the use of pseudonymized information within and outside the institution (Ministry of Health and Welfare of South Korea Dec 2022 ). This paper proposes that the DRB or a data appropriateness review entity comprising researchers, developers, and external members. This entity would review the data collection and management system before commencing healthcare AI research. Such proactive review aims to ensure the safety, appropriateness, feasibility, and absence of biases in data utilization for healthcare AI research.

Stage 3 involves the selection and preliminary assessment of algorithms, making the initiation of full-scale research. At this stage, researchers and developers undergo an IRB review encompassing all facets of conducting the study. They are required to provide extensive justifications concerning the study’s objectives, data standardization, and potential biases. The IRB, compared to the DRB, evaluates the appropriateness of the algorithm, the predictability and validity of results based on initial dataset, the reliability and safety of data management, and ensures the unbiased use of algorithm and data. Researchers, for reporting their conduct to the IRB, should consistently refer to the ethical framework established in stage 1. Considering that data utilization might vary concerning the algorithm used, distinct review rules are set by the DRBs and the IRBs. The former focuses on data management practice, while the latter oversees data utilization practices. This stage functions similar to phase I where the accuracy and appropriateness of the algorithm are determined and reviewed based on validated preliminary data. It can be paralleled with phase I safety assessments in clinical trials, wherein the interaction of an experimental medical device or drug with the human body is examined based on a small number of research subjects.

Stage 4 encompasses the training, validation, and evaluation of the algorithm using the collected real-world data. The training data should be divided into train and test sets, and a pre-prepared validation set, distinct from the training data, is essential for validating healthcare AI algorithms to prevent overfitting and assess real-world applicability. The management of the validation process is imperative to avoid the exportation of models that are only useful during the training process to the actual application phase. and it is recommended that the research and development organization check this process. In the context of healthcare AI applications such as diagnostic imaging, patient risk prediction, and personalized treatment planning, each employing base algorithms ranging from deep learning to decision trees, the need for tailored validation processes becomes clear. For diagnostic imaging or patient risk prediction models, the validation process should primarily focus on rigorous statistical evaluation to ensure accuracy and reliability. Personalized treatment planning systems necessitate validation that emphasizes clinical relevance and the improvement of patient outcomes. These validation processes are essential for assessing the reliability of healthcare AI models. This stage can be seen as akin to phase II in clinical research, the phase that evaluates the effectiveness of a medical device or drug against a placebo. The emphasis is particularly placed on validating the trained algorithm and its relevance to clinical procedures.

Stage 5 involves the deployment of the developed healthcare AI algorithm into practical settings. The regulatory landscape governing healthcare AI implementation may vary based on its real-world application within a country. In South Korea, for example, AI model is evaluated and approved as a medical device by the Ministry of Food and Drug Safety. Moreover, for seeking for the National Health Insurance reimbursement, assessing safety, effectiveness, and economic evaluation from responsible regulators are mandatory. Throughout the step, the organization requires to pursue external validation for its development process, algorithms, and applications while prioritizing transparency. Furthermore, since the nature of healthcare AI includes continuous learning and development as part of its attributes, stage 5 also assigns responsibility for ongoing monitoring, identifying the entity accountable for managing the model. This stage corresponds to phase III, large trials, in clinical trials, where large-scale RCTs are used to determine the applicability of a treatment or new drug, in terms of determining the real-world applicability of a healthcare AI algorithm and putting it to work in the field.

Stage 6 mandates all parties involved to review the process of the continued deployment and ongoing development once the developed algorithm or model has been put into operation in a healthcare setting. Continuous review of use of the model and the functionality of the ethical framework remains pivotal. Maintaining transparent and collaborative communication among all stakeholders emerges as a necessity. In addition, vigilant monitoring of ongoing evolution of the model is imperative to prevent that decision-making based on real-world data might lead to unintended harms. This phase emphasized the follow-up and surveillance of algorithms and models post-launch, analogous to phase IV, post-market Surveillance in clinical research, which refers to the follow-up phase after clinical implementation of a medical device or drug.

The six-stage healthcare AI research governance proposed in this study can be compared to the five-phase standard, BS30440, recently proposed by the UK (Sujan et al. 2023 ). Set to take effect in the second quarter of 2023, BS30440 provides guidelines for validating AI systems in healthcare in the UK context. The guidelines reflect the product life-cycle of healthcare AI, which consists of inception, development, validation, deployment, and monitoring. Compared to the UK guidelines, the six stages presented in this paper add a preliminary ethical framework design and committee verification of data collection and management, distinguishing stages between algorithm determination and subsequent training, validation, and evaluation. BS30440 lacks stipulations for preliminary procedures or data management, integrates algorithm determination and training as a singular process, and makes validation as a separate process. Notably, our study’s governance procedure is designed to extend existing clinical research management procedures, whereas BS30440 establishes novel procedures. This study only examines these distinctions not to favour one framework over the other but to underscore the global development and application of similar governance procedures, extending beyond South Korea.

Limitations and Future Research

The governance guidelines bear inherent limitations. Foremost, they do not decisively address the liability associated with possible harm resulting from healthcare AI applications. In the case of healthcare AI research and application involving multiple parties, it is necessary to examine whether the harm caused can be assumed the same as the existing medical liability process. For example, if a patient is physically harmed in the process of utilizing a healthcare AI device, but it turns out to be a problem with the algorithm rather than the fault of the medical practitioner or device user, who should be held liable?

Navigating liability questions amidst the overlapping influences of various actors poses challenges (Kim 2017 ). While the governance of healthcare AI research needs to address the issue of liability, it is limited by the fact that the guidelines in the study focus on proposing an ethical model grounded in self-regulation, addressing the intricacies of liability remains a significant challenge. Moreover, the procedures are set to be adjusted according to each country’s regulatory procedures, which is because the procedures correspond to existing clinical research guidelines, but it is necessary to examine whether they can be properly operated in real-life situations. This is an area that requires empirical verification by applying the guidelines to actual healthcare AI research governance. Therefore, this paper calls for further research on the healthcare AI governance guidelines presented here to address the issues identified above, especially linking it the legal standard to regulation.

To address the identified limitations, researchers are actively engaged in ongoing efforts in education of AI researchers and the public, social communication, and the revision of the guidelines. These initiatives will ensure a comprehensive societal understanding and adoption of healthcare AI research ethics, encourage researchers and developers to accept the need to conduct research ethically, and thereby facilitate the operationalization of ethical governance systems at both institutional and national levels within the South Korean context. As a result, these endeavours will significantly contribute to the establishment of a robust ethical normative framework for healthcare AI research in this country.

Since the governance settings presented in this study are from the perspective of a specific country, it is necessary to collect the opinions of researchers and bioethicists from other countries through international discussions and reviews. In order to facilitate such discussions, this study aims to inform other countries about the governance system established in South Korea and, using this study as a starting point, collect multi-perspective and multi-disciplinary views on healthcare AI research governance that have not yet been organized and provide basic data on the establishment of cross-border healthcare AI research governance.

The aims of this study are to present a healthcare AI research governance system founded on the South Korean ‘Research Ethics Guidelines for AI Researchers in Healthcare’ and to elucidate each procedural step. The six-stage healthcare AI research governance framework mirrors the healthcare AI research and development process, and is designed in harmony with the existing clinical research management systems. This parallel structure facilitates the utilization of established research management resources and foster mutual understanding among researchers and institutions for conducting ethical research procedures. Nonetheless, the guidelines are likely to reflect the specificities of the Korean healthcare environment, emphasizing the need for further international dialogue and refinement.

Data Availability

The framework employed in our research is included in the English version of “Research Ethics Guidelines for Healthcare AI Researchers” (KNIH 2023 ). This document is currently in the process of being published. Upon its publication, we will promptly provide the relevant link.

Cross-Industry Standard Process for Data Mining

Knowledge Discovery in Database

Sample, Explore, Modify, Model, and Assess

Cognitive Project Management for AI

Cha, Hyun-Jae., and Junhewk Kim. 2022. The ethical approach to health data donation and sharing: From the process of human tissue donation. Bio, Ethics and Policy 6 (2): 101–137.

Google Scholar  

Gerke, Sara, Boris Babic, I. Theodoros Evgenious, and Glenn Cohen. 2020. The need for a system view to regulated artificial intelligence/machine learning-based software as medical device. NPJ Digital Medicine  3: 53. https://doi.org/10.1038/s41746-020-0262-2 .

Higgins, David, and Vince I. Madai. 2020. From bit to bedside: A practical framework for artificial intelligence product development in healthcare. Advanced Intelligent Systems 2 (10): 2000052. https://doi.org/10.1002/aisy.202000052 .

Article   Google Scholar  

Kim, Junhewk. 2017. Autonomous decision medical system and moral responsibility. Philosophy of Medicine 24: 147–182.

Kim, Hannah, Jung Im Lee, Jinah Sim, Yuri Lee, So Yoon Kim, Eun-Ae Kim, Soo Min Kim, and Junhewk Kim. 2023. Ethical guidelines for artificial intelligence research in healthcare: Introducing South Korean Perspectives. Korean Journal of Medicine and Law 31 (1): 85–110. https://doi.org/10.17215/kaml.2023.06.31.1.85 .

Korean National Institute of Health. 2023. Research ethics guidelines for AI researchers in healthcare . Cheongju: Korea Disease Control and Prevention Agency.

Leonelli, Sabina. 2016. Data-centric biology: A philosophical study . Chicago: The University of Chicago Press.

Book   Google Scholar  

Li, Hong, Gustavo Glusman, Hao Hu, Shankaracharya, Juan Caballero, and Robert Hubley, et al. 2014. Relationship estimation from whole-genome sequence data. PLoS Genetics 10(1): e1004144. http://dx.doi.org/10.1371/journal.pgen.1004144.

Metcalf, Jacob, and Kate Crawford. 2016. Where are human subjects in Big Data research? The emerging ethics divide. Big Data & Society 3 (1): 1–14. https://doi.org/10.1177/2053951716650211

Ministry of Food and Drug Safety of the Republic of Korea. 2023. 2022 Medical device license report . Cheongju: Ministry of Food and Drug Safety of the Republic of Korea.

Ministry of Health and Welfare of South Korea. 2022. Guidelines for utilization of healthcare data . Sejong: Ministry of Health and Welfare of South Korea.

National Commission for the Protection of Human Subjects of Biomedical & Behavioral Research. 1978. The Belmont report: Ethical principles and guidelines for the protection of human subjects of research . Bethesda, MD: National Commission for the Protection of Human Subjects of Biomedical & Behavioral Research.

Pianykh, Oleg S., Georg Langs, Marc Dewey, Dieter R. Enzmann, Christian J. Herold, and Stefan O. Schoenberg, et al. 2020. Continuous learning AI in radiology: Implementation principles and early application. Radiology 297 (1): 6–14. https://doi.org/10.1148/radiol.2020200038 .

Rajpurkjar, Pranav, Emma Chen, Oishi Banerjee, and Eric J. Topol. 2022. AI in health and medicine. Nature Medicine 28: 31–38. https://doi.org/10.1038/s41591-021-01614-0 .

Sujan, Mark, Cassius Smith-Frazer, Christina Malamateniou, Joseph Connor, Allison Garner, Harriet Unsworth, et al. 2023. Validation framework for the use of AI in healthcare: Overview of the new British standard BS30440. BMJ Health & Care Informatics 30: e100749. https://doi.org/10.1136/bmjhci-2023-100749 .

Throne, Robin. 2022. Adverse trends in data ethics: The AI Bill of Rights and Human Subjects Protections. SSRN , 30 November 2022. https://doi.org/10.2139/ssrn.4279922 .

Wu, Eric, Kevin Wu, Roxana Daneshjou, David Ouyang, Daniel E. Ho, and James Zou. 2021. How medical AI devices are evaluated: Limitations and recommendations from an analysis of FDA approvals. Nature Medicine 27: 582–584. https://doi.org/10.1038/s41591-021-01312-x .

Download references

Acknowledgements

The authors wish to thank Dr Jung-Im Lee and Dr Sumin Kim for their contribution in developing the guidelines. The first project (2022-ER0807-00) conducted consultation meetings of two panels of interdisciplinary expert participants from law, public health policy, ethics, AI, and patients group for four times from August, 2022, to February, 2023, and a public hearing at February 2023. We deeply express our gratitude for all participants for their valuable opinions.

This work was supported by the ‘Development of Ethics Guidelines and Education Program for the Use of Artificial Intelligent in Healthcare Research’ and ‘Operation of Education Program and Improvement of Ethics Guidelines for the Use of Artificial Intelligent in Healthcare Research’ from the Korean National Institutes of Health (Grant numbers: 2022-ER0807-00 and 2023-ER0808-00).

Author information

Authors and affiliations.

Department of Dental Education, College of Dentistry, Yonsei University, Seoul, South Korea

Junhewk Kim

Asian Institute for Bioethics and Health Law, Department of Medical Humanities and Social Sciences, College of Medicine, Yonsei University, Seoul, South Korea

So Yoon Kim & Hannah Kim

Center for Research Compliance, Ewha Womans University, Seoul, South Korea

Department of AI Convergence, Hallym University, Chuncheon, South Korea

Department of Health and Medical Information, Myongji College, Seoul, South Korea

You can also search for this author in PubMed   Google Scholar

Contributions

J. K. and H. K. were responsible for the conception, design, acquisition of data or analysis, and interpretation of data. J. K. was responsible for manuscript writing, subsequent revisions of the manuscript and funding (2023-ER0808-00). H. K. was responsible for reviewing the manuscript, funding (2022-ER0807-00), and developing the guidelines. S. Y. K., E. A. K., J. A. S., and Y. L. participated in developing the guidelines and reviewing the manuscript. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Hannah Kim .

Ethics declarations

During the research, the main responsibilities of the funding agency included managing the project progress and making decisions regarding the publication of the guidelines and the agency had no role in the study design, data collection and analysis, preparation of the manuscript, and decision to publish it.

Ethics Approval

As this study did not involve human participants, ethics approval was not needed.

Consent to Participate

Not applicable.

Consent for Publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Kim, J., Kim, S.Y., Kim, EA. et al. Developing a Framework for Self-regulatory Governance in Healthcare AI Research: Insights from South Korea. ABR (2024). https://doi.org/10.1007/s41649-024-00281-w

Download citation

Received : 30 September 2023

Revised : 11 January 2024

Accepted : 12 January 2024

Published : 25 March 2024

DOI : https://doi.org/10.1007/s41649-024-00281-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Healthcare AI research
• Research ethics
• Research governance
• AI governance
• Self-regulation
• Interdisciplinary approach
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 21 March 2024

GRADE-ADOLOPMENT of hyperthyroidism treatment guidelines for a Pakistani context

• Russell Seth Martins 1   na1 ,
• Sarah Nadeem 1 , 2 , 3   na1 ,
• Abeer Aziz 1 ,
• Sajjan Raja 4 ,
• Alina Pervez 1 ,
• Najmul Islam 2 ,
• Asma Ahmed 2 ,
• Aisha Sheikh 2 ,
• Saira Furqan 2 ,
• Nanik Ram 2 ,
• Azra Rizwan 2 ,
• Nashia Ali Rizvi 1 ,
• Mohsin Ali Mustafa 1 ,
• Salima Saleem Aamdani 5 ,
• Bushra Ayub 6 &
• Muhammad Qamar Masood 2  

BMC Endocrine Disorders volume  24 , Article number:  41 ( 2024 ) Cite this article

116 Accesses

1 Altmetric

Metrics details

Introduction

The prevalence of hyperthyroidism in Pakistan is 2.9%, which is two times higher than in the United States. Most high-quality hyperthyroidism clinical practice guidelines (CPGs) used internationally originate from high-income countries in the West. Local CPGs in Pakistan are not backed by transparent methodologies. We aimed to produce comprehensive, high-quality CPGs for the management of hyperthyroidism in Pakistan.

We employed the GRADE-ADOLOPMENT approach utilizing the 2016 American Thyroid Association Guidelines for Diagnosis and Management of Hyperthyroidism and Other Causes of Thyrotoxicosis as the source CPG. Recommendations from the source guideline were either adopted as is, excluded, or adapted according to our local context.

The source guideline included a total of 124 recommendations, out of which 71 were adopted and 49 were excluded. 4 recommendations were carried forward for adaptation via the ETD process, with modifications being made to 2 of these. The first addressed the need for liver function tests (LFTs) amongst patients experiencing symptoms of hepatotoxicity while being treated with anti-thyroid drugs (ATDs). The second pertained to thyroid status testing post-treatment by radioactive iodine (RAI) therapy for Graves’ Disease (GD). Both adaptations centered around the judicious use of laboratory investigations to reduce costs of hyperthyroidism management.

Our newly developed hyperthyroidism CPGs for Pakistan contain two context-specific modifications that prioritize patients’ finances during the course of hyperthyroidism management and to limit the overuse of laboratory testing in a resource-constrained setting. Future research must investigate the cost-effectiveness and risk-benefit ratio of these modified recommendations.

Peer Review reports

Hyperthyroidism is a common endocrine condition that presents a significant global challenge with high morbidity and mortality rates [ 1 , 2 ]. In Pakistan, a South Asian lower-middle-income country (LMIC) with a population of over 220 million [ 3 ], the prevalence is 2.9% [ 4 ]. This is more than two times higher than the United States of America (US: 1.3% [ 5 ]) and more than three times higher than in Europe (0.8% [ 6 ]). The high prevalence of hyperthyroidism in Pakistan can be attributed to a complex interplay of factors, with key determinants including geographical variables and ethnic diversity [ 7 ]. Hyperthyroidism confers an increased all-cause mortality risk, particularly due to cardiovascular causes [ 8 ].

Evidence-based clinical practice guidelines (EBCPGs) direct the diagnosis and management of hyperthyroidism, so as to achieve standardization of favorable clinical outcomes [ 9 , 10 ]. EBCPGs created by institutions in developed countries in the West, such as the US [ 11 ] and European countries [ 12 ], are oftentimes adopted by other countries, particularly LMICs, for use in their settings. This is because LMICs, like Pakistan, usually lack the research infrastructure and financial resources to independently develop EBCPGs de novo for their own healthcare context [ 13 ]. However, the application of such EBCPGs for the management of hyperthyroidism in Pakistan presents a problem, as the country’s landscape differs due to several factors [ 14 ]. These include disease epidemiology [ 15 ], healthcare financing [ 16 ], dietary habits and iodine consumption [ 17 ], socio-economic influences [ 18 ], and disease-related awareness [ 19 ]. Therefore, it becomes imperative for an LMIC like Pakistan to create EBCPGs that best suit the unique context of the setting where they will be applied.

In cases where the de novo creation of EBCPGs is not practically feasible, a process called “adolopment” provides a suitable alternative. Adolopment describes a combination of adoption (verbatim use), adaptation (contextual modifications), and de novo development, thus leveraging the benefits of pre-existing high-quality EBCPGs while ensuring local appropriateness. The GRADE-ADOLOPMENT method [ 13 ], developed by GRADE (Grading of Recommendations Assessment, Development, and Evaluation), is a globally accepted and implemented process of EBCPG adolopment. It uses evidence-to-decision (ETD) tables, which summarize the best available evidence on a topic, to guide decisions regarding the need for contextual modifications of individual recommendations within an EBCPG [ 20 ]. GRADE-ADOLOPMENT has been used in countries and regions across the world, including Saudi Arabia [ 13 ], Australia [ 21 ], Tunisia [ 22 ], the Eastern Mediterranean region [ 23 ], the Asia-Pacific region [ 24 ], Mexico [ 25 ], and the United Kingdom [ 26 ].

Although the Pakistan Endocrine Society, founded in 2003, is involved in the creation of local EBCPGs for the management of common endocrine disorders in Pakistan, their publications have thus far focused on diabetes mellitus and metabolic syndrome [ 27 ]. Moreover, the processes involved in the development of these EBCGPs are not explicitly described. Consequently, there is immense need for local hyperthyroidism EBCPGs to be developed following a transparent, standardized process that makes use of existing available best-evidence EBCPGs with appropriate context-specific modifications. Such EBCPGs would bring the healthcare system of Pakistan a step closer to achieving optimal health outcomes in hyperthyroidism and would gain credibility by virtue of their transparent development processes. Thus, we aimed to employ the GRADE-ADOLOPMENT process to develop local evidence-based EBCPGs for the management of hyperthyroidism in adults by GPs in Pakistan.

Methodology

This process was conducted at the CITRIC (Clinical and Translational Research Incubator) Center for Clinical Best Practices (CCBP) at the Aga Khan University (AKU), Pakistan. The AKU is a private sector, not-for‐profit hospital in Pakistan, and is also the country’s leading healthcare and biomedical research facility [ 28 ].

The CITRIC CCBP at AKU is tasked with the adaptation and development of EBCPG and care pathways to standardize and improve healthcare in Pakistan. The GRADE-ADOLOPMENT processes described in this study have been implemented by the CCBP, in collaboration with the expertise of the Section of Endocrinology at AKU and the GRADE-USA working group, in the development of hyperthyroidism management EBCPGs for use by general practitioners (GPs)/primary care physicians in Pakistan. The decision to create hyperthyroidism EBCPGs for GPs rather than specialist endocrinologists is due to the lack of access to specialists in Pakistan [ 29 ].

The study team is comprised of the CCBP research staff (who are proficient in GRADE methodology and in the development of EBCPGs) as well as endocrinology faculty led by Endocrinology Section Head of AKU.

Source guideline selection

The source guideline is the single, original, “parent” EBCPG that undergoes the GRADE-ADOLOPMENT process in the development of a local EBCPG. The 2016 American Thyroid Association Guidelines for Diagnosis and Management of Hyperthyroidism and Other Causes of Thyrotoxicosis [ 30 ] was selected by the Section of Endocrinology as the source EBCPG, due to its comprehensive set of recommendations, integrated approach to management, and high-quality synthesis of available evidence. The 2016 American Thyroid Association source guideline used the GRADE approach for the strength of the recommendations and the quality of evidence.

Guideline review

Figure  1 delineates the adolopment process used in our study. First, a Table of Recommendations (ToR) was created by extracting and compiling all recommendations mentioned in the source EBCPG. Two senior attending endocrinologists reviewed the ToR independently and marked each recommendation as either “ Adopt ,” “ Adapt” or “ Exclude .” Discrepancies were settled in consensus with the Section Head of Endocrinology. Recommendations marked “ Adopt ” were incorporated as is or with minor changes into the local EBCPG, while those marked “ Exclude ” were omitted from the local EBCPG. Exclusion was based on the recommendation pertaining to pediatric or inpatient management, or if the recommendation was deemed irrelevant to the local Pakistani context. Other reasons for exclusion were required to be explained by the reviewers.

Recommendations marked “ Adapt ” were deemed to warrant additional review and revision via the GRADE-ADOLOPMENT process (detailed below) before incorporation into the local EBCPGs. Our adolopment process (Fig. 1 ) had two important differentiations to the one described originally [ 13 ]. Firstly, we did not create any recommendations de novo, which was due to a lack of perceived need for additional recommendations. Secondly, recommendations that were deemed to require only minor and straightforward changes prior to adoption were not subjected to the complete adaptation process consisting of ETD tables and expert panel review.

GRADE-ADOLOPMENT process for Hyperthyroidism Management EBCPG for Pakistan

GRADEPro evidence to decision Framework

GRADEPro is a web application used to help create, manage, and share summaries of research evidence [ 31 ]. The CCBP staff involved in this study underwent a training module to master use of GRADEPro for the GRADE-ADOLOPMENT process. The software was used to develop Evidence to Decision (ETD) tables to reach a consensus on each of the recommendations marked “ Adapt .”

ETD tables that summarize evidence to enable members of an expert panel to make healthcare recommendations or decisions. Development of ETD tables begins with formulation of a question structured as follows: “Should the Intervention/Suggested Change be favored over the Comparison/Current Standard of Practice? ” The pros and cons of the suggested change are judged by an expert panel across 12 criteria, that are shown in Supplementary Table 1 (Additional file 1 ).

Each criterion was supported with evidence gathered through a best evidence review process (Additional file 1 ), to provide local context for the pros and cons of the recommendation. The CCBP team summarized the newly gathered evidence for each criterion in the “ Research Evidence ” and “ Additional Considerations ” columns. The GRADE-USA working group was deeply involved in the creation of the ETD tables.

Expert panel review

An expert panel of five endocrinology faculty from AKU were invited by the Endocrinology Section Head to review the completed ETD table for each recommendation and provide their judgement for each criterion. This judgment was in the form of a single selection from multiple response options. If, for any criteria, an expert required additional evidence, they informed the CCBP team. An effort was made to source the requisite information, which, if found, was shared with all the panel members. Experts’ judgements were sought in an anonymous and confidential manner, with the GRADEPro software allowing reviewers to select options and provide feedback without their identity known to fellow experts or the CCBP team. A sample of a GRADEPro ETD is shown as Supplementary Table 2 (Additional file 1 ).

Final recommendation revisions & synthesis

Once all the members of the expert panel had provided their responses to the ETD, the CCBP staff synthesized their responses to produce a summary of judgments. The CCBP staff conducted a meeting with the expert panel to review the summary of judgments and reach a final unanimous consensus on the need for and nature of any revisions to the recommendations in question. The strength of each recommendation was also decided. Finally, the consensus was presented to the Section Head of Endocrinology for review, after which the recommendation was incorporated into the Pakistani EBCPG with a summary of the consensus decision.

Final debriefing to identify challenges & explore solutions

Two focus group discussions (FGDs) were conducted to identify challenges faced throughout the entire GRADE-ADOLOPMENT process and to explore corresponding solutions. These FGDs were led by a member of the CCBP team and included the CCBP staff and the Section Head of Endocrinology. Participants were given the opportunity to first brainstorm challenges and solutions independently, and these were then discussed within the FGD. Each challenge was decided as per consensus opinion to be either a major or minor challenge. The CCBP team then categorized the final list of specific challenges within broad themes, and their corresponding solutions were presented alongside them.

Initial review of source guideline

The source guideline included a total of 124 recommendations, out of which 71 were adopted and 49 were excluded. 4 recommendations were carried forward for adaptation via the ETD process (Fig.  2 ) (Supplementary Table 3) (Additional file 1 ). A list of all excluded recommendations can be found in Supplementary Table 4 (Additional file 1 ).

Outcomes of table of recommendations review

Evidence-to-decision (ETD) tables

Amongst the four recommendations that underwent the adaptation process, modifications were made to two (Tables  1 and 2 ), while the remaining two were unchanged (Tables  3 and 4 ). The complete Evidence to Decision tables with the summary of judgements for the modified recommendations can be found in Supplementary Tables 5 & 6 (Additional files 2 & 3 ).

Challenges and solutions

The challenges faced were broadly categorized into four main themes: resources, stakeholder support and involvement, resistance to change, and methodological limitations (Table  5 ). Solutions proposed for the challenges faced will be incorporated in the future updates of the guideline.

In this paper, we applied the GRADE-ADOLOPMENT process to the 2016 American Thyroid Association Guidelines for Diagnosis and Management of Hyperthyroidism and Other Causes of Thyrotoxicosis [ 30 ] to adolop EBCPGs for the management of hyperthyroidism in the local context of Pakistan. Out of a total of 124 recommendations, 71 were adopted, 49 were excluded, and 4 were subjected to the process of adaptation. The adapted recommendations primarily focused on accommodating patient-centered factors and accounting for a lack of resources in Pakistan, without a significant compromise in clinical outcomes.

The first adapted recommendation addressed the need for liver function tests (LFTs) amongst patients experiencing symptoms of hepatotoxicity while being treated with anti-thyroid drugs (ATDs). The source EBCPG recommended a full panel of LFTs (alanine transaminase, aspartate transaminase, alkaline phosphatase, gamma-glutamyl transferase; total, conjugated and unconjugated bilirubin) for all patients experiencing any symptoms remotely suggestive of hepatotoxicity (pruritic rash, jaundice, light-colored stool or dark urine, joint pain, abdominal pain or bloating, anorexia, nausea, or fatigue) [ 30 ]. However, this recommendation was adapted to advise the use of only alanine transaminase (ALT) to diagnose the extent of liver injury in patients experiencing highly specific symptoms (jaundice, pruritis, or change in stool color). The rationale behind this adaptation was centered around the infrequent incidence of hepatotoxicity while on ATD (1.4–6.3% [ 32 ]) and the patient-borne financial ramifications of over-testing. In contrast to high-income countries where government or private insurance covers the majority of healthcare costs, almost 60% of healthcare costs in Pakistan are via out-of-pocket payment by patients [ 33 ], with national health coverage provided to only 20% of the population [ 34 ]. The cost of a full LFT panel in Pakistan ranges from $3.73–7.15, which is between 3 and 7 times more than a single ALT test (ranges from $1.01–1.67). However, while patient finances must be given full consideration in the management of hyperthyroidism, future research is needed to investigate the cost-effectiveness of the adapted recommendation in a Pakistani population.

The second recommendation to undergo the adaptation process was related to thyroid status testing post-treatment by radioactive iodine (RAI) therapy for Graves’ Disease (GD). The source EBCPG recommends assessing free T4 (FT4), total T3, and thyroid-stimulating hormone (TSH) amongst patients within 1–2 months after patients with GD receive RAI therapy, followed by 4–6 weekly testing for 6 months, or until the patient becomes hypothyroid and is stable on thyroid hormone replacement This recommendation was modified to advise the assessment of only FT4 at initial follow-up, with subsequent TSH assessment only in the case of low T4. The keystone of this modification was the consensus that FT4 alone is a sufficiently sensitive modality to detect post-RAI hypothyroidism, and that TSH suppression in the post-RAI period may limit its accuracy in reflecting thyroid status. In fact, this misleading suppression of TSH after RAI therapy may prompt the physician to initiate thioamides unnecessarily. Moreover, in Pakistan, the use of a single FT4 test (ranges from $4.92–8.30) is about a third the price of a full panel consisting of FT4, T3 and TSH ($12.7–18.1). In fact, a sizeable percentage (48.8%) of the overall management costs for hyperthyroidism are attributable to laboratory testing [ 35 ]. Lastly, if both the initial FT4 and subsequent TSH assessment reflect hypothyroidism, and thyroid hormone replacement is initiated and optimized, long-term assessment of treatment effectiveness can be monitored by TSH alone. To facilitate adherence to follow-up and routine post-operative testing, it is recommended that public and private laboratories in Pakistan should partner with healthcare centers to create comprehensive and appropriate care packages which include all post-treatment management and surveillance.

The third recommendation that underwent the adaptation process concerned the preoperative administration of potassium iodide (KI; Lugol’s solution), in addition to ATD and/or beta-blockers, prior to surgical management of GD. While no changes were enacted to this recommendation, experts noted that KI was not widely accessible in Pakistan, with availability of KI being restricted to tertiary care hospitals and large-scale pharmacies, even in urban settings. Though the supporting evidence lacks robustness and clarity, KI is believed to limit intraoperative blood loss by decreasing thyroid vascularity, and also suppress the synthesis and release of thyroid hormone [ 11 ]. However, despite these benefits, the lack of widespread availability of KI in Pakistan would undoubtedly preclude its universal use before surgery for GD. Moreover, recent studies have once again called into question the benefits of preoperative KI administration, with regards to its impacts on intraoperative bleeding, difficulty of operation, operative time, and postoperative outcomes [ 36 , 37 , 38 ]. Thus, the expert team added an additional comment after adopted recommendation, which reassured readers that a lack of administration of KI would likely not compromise the health outcomes of a patient.

The final recommendation that underwent the adaptation process advised a single dose of RAI to render a patient with GD hypothyroid. Although no modifications were made to the recommendation, discussions centered around the cost-effectiveness and availability of RAI versus an alternate option of employing ATD therapy with regular thyroid function test (TFT) monitoring. However, though ATD therapy may provide a more financially feasible mode of treatment, the remission rate of GD with RAI therapy is significantly higher than with ATD therapy [ 35 ]. Therefore, RAI should be considered for definitive treatment in GD patients on high doses of ATD treatment, those not responding to the ATD treatment, and those requiring ATD treatment for more than 2 years.

There are limitations to the GRADE-ADOLOPMENT process used in our study that we would like to acknowledge. Firstly, individual-level (e.g., the Section Head reviewing each ToR to independently to decide whether to adopt, adapt or exclude recommendations) and group-level (e.g., the consensus meeting featuring five experts from a single institution) biases may limit the applicability of our EBCPGs to other settings in Pakistan. Additionally, fundamental to the GRADE-ADOLOPMENT process, the adaptation process was guided primarily by expert consensus, due to the suboptimal availability of local, high-quality level of evidence. Moreover, we limited the inclusion of other important stakeholders, such as patients, allied health professionals, general practitioners, nurses, experts external to AKU, other healthcare centers, external endocrinology organizations or societies, and provincial and federal governments. This was to minimize inevitable delays that would have accompanied a larger team, including logistic difficulties, conflicts of interest, lack of mutual availability, political influences, and lack of direct incentives. However, prior experience in developing such EBCPGs enabled the CCBP team to remain mindful of the needs and values of these groups to a large extent. Lastly, while the efforts to create a local EBCPG for the management of hyperthyroidism have yielded success, the feasibility of widespread utilization and implementation of the EBCPG across Pakistan remains a concern. All the aforementioned limitations represent real-world barriers to the idealistic implementation of the GRADE-ADOLOPMENT process in resource-constrained and poorly structured healthcare systems in LMICs like Pakistan.

The outcome of the GRADE-ADOLOPMENT process applied to the 2016 American Thyroid Association Guidelines for Diagnosis and Management of Hyperthyroidism and Other Causes of Thyrotoxicosis [ 30 ] yielded two major changes in the newly developed Pakistani EBCPG for the management of hyperthyroidism. These included the recommendation to assess only ALT (as opposed to a full LFT panel) amongst patients on ATDS experiencing symptoms highly specific of hepatotoxicity (as opposed to a higher index of suspicion considering non-specific symptoms like bloating, anorexia, nausea, or fatigue), and the recommendation to assess only FT4 (as opposed to the full panel of FT4, total T3, and TSH) at initial follow-up after RAI therapy for GD, with subsequent TSH assessment only in the case of low T4. The rationale behind both these changes were to prioritize patients’ finances during the course of hyperthyroidism management and to limit the overuse of laboratory testing in a resource-constrained setting. Future research must investigate the cost-effectiveness and risk-benefit ratio of these modified recommendations.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.

Taylor PN, Albrecht D, Scholz A, Gutierrez-Buey G, Lazarus JH, Dayan CM, et al. Global epidemiology of hyperthyroidism and hypothyroidism. Nat Rev Endocrinol. 2018;14(5):301–16.

Article   PubMed   Google Scholar  

Devereaux D, Tewelde SZ. Hyperthyroidism and thyrotoxicosis. Emerg Med Clin. 2014;32(2):277–92.

Article   Google Scholar  

Initiatives MoPDS. Pakistan population situation analysis. Pakistan: UNFPA; 2020.

Akhter S, Khan A, Siddiqui MM, Nawab G. Frequencies of thyroid problems in different age, sex and seasons. J Med Sci. 2001;1:153–6.

Hollowell JG, Staehling NW, Flanders WD, Hannon WH, Gunter EW, Spencer CA, et al. Serum TSH, T(4), and thyroid antibodies in the United States population (1988 to 1994): National Health and Nutrition Examination Survey (NHANES III). J Clin Endocrinol Metab. 2002;87(2):489–99.

Article   CAS   PubMed   Google Scholar  

Garmendia Madariaga A, Santos Palacios S, Guillén-Grima F, Galofré JC. The incidence and prevalence of thyroid dysfunction in Europe: a meta-analysis. J Clin Endocrinol Metab. 2014;99(3):923–31.

Jawa A, Jawad A, Riaz SH, Assir MZ, Chaudhary AW, Zakria M, et al. Turmeric use is associated with reduced goitrogenesis: thyroid disorder prevalence in Pakistan (THYPAK) study. Indian J Endocrinol Metab. 2015;19(3):347–50.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Selmer C, Olesen JB, Hansen ML, von Kappelgaard LM, Madsen JC, Hansen PR, et al. Subclinical and overt thyroid dysfunction and risk of all-cause mortality and cardiovascular events: a large population study. J Clin Endocrinol Metab. 2014;99(7):2372–82.

Haymart MR. The role of clinical guidelines in patient care: thyroid hormone replacement in women of reproductive age. Thyroid. 2010;20(3):301–7.

Article   PubMed   PubMed Central   Google Scholar  

Mestman JH. How effective are clinical guidelines for hypothyroidism in pregnancy in clinical practice? Clin Thyroidol. 2013;25:194–6.

Ross DS, Burch HB, Cooper DS, Greenlee MC, Laurberg P, Maia AL, et al. 2016 American Thyroid Association guidelines for diagnosis and management of hyperthyroidism and other causes of thyrotoxicosis. Thyroid. 2016;26(10):1343–421.

Kahaly GJ, Bartalena L, Hegedüs L, Leenhardt L, Poppe K, Pearce SH. 2018 European thyroid Association Guideline for the management of Graves’ hyperthyroidism. Eur Thyroid J. 2018;7(4):167–86.

Schünemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, Mustafa RA, Manja V, et al. GRADE evidence to decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017;81:101–10.

De Leo S, Lee SY, Braverman LE. Hyperthyroidism. Lancet. 2016;388(10047):906–18.

Alam¹ Z, Shah M, Khan M, Ali W, Shehzad A, Shah JA, et al. Thyroid dysfunction and prevalence of both clinical and subclinical form of hyperthyroidism and hypothyroidism in District Mardan, KPK, Pakistan. 2019;8:98–104.

Alvi AM, Azmat U, Shafiq W, Rasheed AHA, Siddiqi AI, Khan S, et al. Efficacy of radioiodine therapy in patients with primary hyperthyroidism: an institutional review from Pakistan. Cureus. 2022;14(5).

Sarwar S. Iodized salt: a risk factor for hyperthyroidism. J Rawalpindi Med Coll. 2013;17(2):284–7.

ADS   Google Scholar  

Olmos RD, De Figueiredo R, Aquino E, Lotufo P, Bensenor IM. Gender, race and socioeconomic influence on diagnosis and treatment of thyroid disorders in the Brazilian Longitudinal Study of Adult Health (ELSA-Brasil). Braz J Med Biol Res. 2015;48:751-8.

Shaffique S, Anwar H, Asif HM, ul Haq I, Akram MJRJP, Sciences P. Prevalence of hyperthyroidism and its impact on Quality of Life among students of the Islamia. Univ Bahawalpur Pakistan. 2020;8(2):85–90.

Google Scholar  

Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE evidence to decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. Clin Pract Guidelines. 2017;2(2):167.e1-e10.

Okely AD, Ghersi D, Hesketh KD, Santos R, Loughran SP, Cliff DP, et al. A collaborative approach to adopting/adapting guidelines - the Australian 24-hour movement guidelines for the early years (birth to 5 years): an integration of physical activity, sedentary behavior, and sleep. BMC Public Health. 2017;17(Suppl 5):869.

Kahale LA, Ouertatani H, Brahem AB, Grati H, Hamouda MB, Saz-Parkinson Z, et al. Contextual differences considered in the Tunisian ADOLOPMENT of the European guidelines on breast cancer screening. Health Res Policy Syst. 2021;19(1):80.

Darzi A, Harfouche M, Arayssi T, Alemadi S, Alnaqbi KA, Badsha H, et al. Adaptation of the 2015 American College of Rheumatology treatment guideline for rheumatoid arthritis for the Eastern Mediterranean Region: an exemplar of the GRADE Adolopment. Health Qual Life Outcomes. 2017;15(1):183.

Loo BKG, Okely AD, Pulungan A, Jalaludin MY. Asia-Pacific Consensus Statement on integrated 24-hour activity guidelines for children and adolescents. 2022;56(10):539 − 45.

Coronado R, Gómez de León AO, Faba-Beaumont M. Adaptation of clinical practice guidelines for osteoporosis in a Mexican context. Experience using methodologies ADAPTE, GRADE-ADOLOPMENT and RAND/UCLA. J Clin Epidemiol. 2020;131:30–42.

Reilly JJ, Hughes AR, Janssen X, Hesketh KR, Livingstone S, Hill C, et al. GRADE-ADOLOPMENT process to develop 24-hour movement behavior recommendations and physical activity guidelines for the under 5s in the United Kingdom, 2019. J Phys Act Health. 2020;17(1):101–8.

Society PE. Publication & trials. Available from: https://www.pakendosociety.org/publications-trials/ .

Haq IU, Rehman ZU. Medical Research in Pakistan; A Bibliometric Evaluation from 2001 to 2020. Libr Philos Pract. 2021:1–13.

Bhatti MW. Country facing shortage of internal medicine specialists owing to non-practising female doctors. Pakistan: The News International; 2022. thenews.com.pk .

RossDouglas S, BurchHenry B, CooperDavid S, Carol G, Luiza M, RivkeesScott A, et al. 2016 American Thyroid Association guidelines for diagnosis and management of hyperthyroidism and other causes of thyrotoxicosis. Thyroid. 2016;26(10):1343–1421.

GRADEpro G, GRADEpro GDT. GRADEpro Guideline Development Tool [Software]. McMaster University, 2015 (developed by Evidence Prime, Inc.); 2015.

Suzuki N, Noh JY, Hiruma M, Kawaguchi A, Morisaki M, Ohye H, et al. Analysis of antithyroid drug-Induced severe liver Injury in 18,558 newly diagnosed patients with Graves’ disease in Japan. Thyroid. 2019;29(10):1390–8.

Statistics PBo. Government of Pakistan Bureau of Statistics Karachi. 2021. Available from: https://www.pbs.gov.pk .

Cheema AR, Zaidi S, Najmi R, Khan FA, Kori SA, Shah NA. Availability does not mean utilisation: analysis of a large micro health insurance programme in Pakistan. Global J Health Sci. 2020;12(10):1–4.

Zuberi LM, Awan S, Islam N, Akhter J, Jabbar A. Does choice of therapy save costs and improve outcomes in hyperthyroid patients. J Pak Med Assoc. 2008;58(6):309.

PubMed   Google Scholar  

Lindner K, Kußmann J, Fendrich V. Preoperative potassium iodide treatment in patients undergoing thyroidectomy for Graves’ disease—perspective of a European high-volume center. World J Surg. 2020;44(10):3405–9.

Randle RW, Bates MF, Long KL, Pitt SC, Schneider DF, Sippel RS. Impact of potassium iodide on thyroidectomy for Graves’ disease: implications for safety and operative difficulty. Surgery. 2018;163(1):68–72.

Kalra S, Kalra B. Mishti copes with diabetes: a pragmatic approach to coping skills training. J Soc Health Diabetes; New York. 2015;5(1):1–2.

Download references

Acknowledgements

Author information.

Russell Seth Martins and Sarah Nadeem have contributed equally to this manuscript and wish to be considered as joint first authors.

Authors and Affiliations

Center for Clinical Best Practices, Clinical and Translational Research Incubator (CITRIC), Aga Khan University, Karachi, 74800, Pakistan

Russell Seth Martins, Sarah Nadeem, Abeer Aziz, Alina Pervez, Nashia Ali Rizvi & Mohsin Ali Mustafa

Section of Endocrinology, Department of Medicine, Aga Khan University, Karachi, 74800, Pakistan

Sarah Nadeem, Najmul Islam, Asma Ahmed, Aisha Sheikh, Saira Furqan, Nanik Ram, Azra Rizwan & Muhammad Qamar Masood

FACE (Fellow American College of Endocrinology), Internal Medicine & Endocrinology, Diabetes & Metabolism, Internal Medicine, and Endocrinology, Women in Medicine Committee, Associate Dean’s Women Faculty Forum, Aga Khan University, Karachi, Pakistan

Sarah Nadeem

Medical College, Aga Khan University, Karachi, 74800, Pakistan

Sajjan Raja

Department of Medicine, Aga Khan University, Karachi, 74800, Pakistan

Salima Saleem Aamdani

Learning Research Centre, Patel Hospital, Karachi, 75300, Pakistan

Bushra Ayub

You can also search for this author in PubMed   Google Scholar

Contributions

RSM, NI, AAh, AS, SF, NR, AR, NAR, MAM, SSA, BA, SN and MQM were involved in the conceptualization of the manuscript. NI, AAh, AS, SF, NR, AR, NAR, MAM, SSA, BA, SN and MQM were involved in the GRADE-ADOLOPMENT of the EBCPGs. RSM, AAz, SR, AP, and SN were involved in the writing of the manuscript. The final draft was reviewed by all authors.

Corresponding author

Correspondence to Sarah Nadeem .

Ethics declarations

Ethics approval and consent to participate.

Given the lack of involvement of patients or other human participants, a waiver of ethics approval and informed consent was obtained from the Ethics Review Committee of the Aga Khan University. All methods were conducted in accordance with the highest ethical standards outlined in the 1964 Declaration of Helsinki and its future amendments.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., additional file 2., additional file 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Martins, R.S., Nadeem, S., Aziz, A. et al. GRADE-ADOLOPMENT of hyperthyroidism treatment guidelines for a Pakistani context. BMC Endocr Disord 24 , 41 (2024). https://doi.org/10.1186/s12902-023-01493-1

Download citation

Received : 31 October 2022

Accepted : 17 October 2023

Published : 21 March 2024

DOI : https://doi.org/10.1186/s12902-023-01493-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

BMC Endocrine Disorders

ISSN: 1472-6823

• General enquiries: [email protected]