research papers chapter 1 to 5

• No category

RESEARCH-PAPER-CHAPTER-1-5 (1)

Related documents

Add this document to collection(s)

You can add this document to your study collection(s)

Add this document to saved

You can add this document to your saved list

Suggest us how to improve StudyLib

(For complaints, use another form )

Input it if you want to receive answer

Research Guide

Chapter 5 sections of a paper.

Now that you have identified your research question, have compiled the data you need, and have a clear argument and roadmap, it is time for you to write. In this Module, I will briefly explain how to develop different sections of your research paper. I devote a different chapter to the empirical section. Please take into account that these are guidelines to follow in the different section, but you need to adapt them to the specific context of your paper.

5.1 The Abstract

The abstract of a research paper contains the most critical aspects of the paper: your research question, the context (country/population/subjects and period) analyzed, the findings, and the main conclusion. You have about 250 characters to attract the attention of the readers. Many times (in fact, most of the time), readers will only read the abstract. You need to “sell” your argument and entice them to continue reading. Thus, abstracts require good and direct writing. Use journalistic style. Go straight to the point.

There are two ways in which an abstract can start:

By introducing what motivates the research question. This is relevant when some context may be needed. When there is ‘something superior’ motivating your project. Use this strategy with care, as you may confuse the reader who may have a hard time understanding your research question.

By introducing your research question. This is the best way to attract the attention of your readers, as they can understand the main objective of the paper from the beginning. When the question is clear and straightforward this is the best method to follow.

Regardless of the path you follow, make sure that the abstract only includes short sentences written in active voice and present tense. Remember: Readers are very impatient. They will only skim the papers. You should make it simple for readers to find all the necessary information.

5.2 The Introduction

The introduction represents the most important section of your research paper. Whereas your title and abstract guide the readers towards the paper, the introduction should convince them to stay and read the rest of it. This section represents your opportunity to state your research question and link it to the bigger issue (why does your research matter?), how will you respond it (your empirical methods and the theory behind), your findings, and your contribution to the literature on that issue.

I reviewed the “Introduction Formulas” guidelines by Keith Head , David Evans and Jessica B. Hoel and compiled their ideas in this document, based on what my I have seen is used in papers in political economy, and development economics.

This is not a set of rules, as papers may differ depending on the methods and specific characteristics of the field, but it can work as a guideline. An important takeaway is that the introduction will be the section that deserves most of the attention in your paper. You can write it first, but you need to go back to it as you make progress in the rest of teh paper. Keith Head puts it excellent by saying that this exercise (going back and forth) is mostly useful to remind you what are you doing in the paper and why.

5.2.1 Outline

What are the sections generally included in well-written introductions? According to the analysis of what different authors suggest, a well-written introduction includes the following sections:

• Hook: Motivation, puzzle. (1-2 paragraphs)
• Research Question: What is the paper doing? (1 paragraph)
• Antecedents: (optional) How your paper is linked to the bigger issue. Theory. (1-2 paragraphs)
• Empirical approach: Method X, country Y, dataset Z. (1-2 paragraphs)
• Detailed results: Don’t make the readers wait. (2-3 paragraphs)
• Mechanisms, robustness and limitations: (optional) Your results are valid and important (1 paragraph)
• Value added: Why is your paper important? How is it contributing to the field? (1-3 paragraphs)
• Roadmap A convention (1 paragraph)

Now, let’s describe the different sections with more detail.

5.2.1.1 1. The Hook

Your first paragraph(s) should attract the attention of the readers, showing them why your research topic is important. Some attributes here are:

• Big issue, specific angle: This is the big problem, here is this aspect of the problem (that your research tackles)
• Big puzzle: There is no single explanation of the problem (you will address that)
• Major policy implemented: Here is the issue and the policy implemented (you will test if if worked)
• Controversial debate: some argue X, others argue Y

5.2.1.2 2. Research Question

After the issue has been introduced, you need to clearly state your research question; tell the reader what does the paper researches. Some words that may work here are:

• I (We) focus on
• This paper asks whether
• In this paper,
• Given the gaps in knoweldge, this paper
• This paper investigates

5.2.1.3 3. Antecedents (Optional section)

I included this section as optional as it is not always included, but it may help to center the paper in the literature on the field.

However, an important warning needs to be placed here. Remember that the introduction is limited and you need to use it to highlight your work and not someone else’s. So, when the section is included, it is important to:

• Avoid discussing paper that are not part of the larger narrative that surrounds your work
• Use it to notice the gaps that exist in the current literature and that your paper is covering

In this section, you may also want to include a description of theoretical framework of your paper and/or a short description of a story example that frames your work.

5.2.1.4 4. Empirical Approach

One of the most important sections of the paper, particularly if you are trying to infer causality. Here, you need to explain how you are going to answer the research question you introduced earlier. This section of the introduction needs to be succint but clear and indicate your methodology, case selection, and the data used.

5.2.1.5 5. Overview of the Results

Let’s be honest. A large proportion of the readers will not go over the whole article. Readers need to understand what you’re doing, how and what did you obtain in the (brief) time they will allocate to read your paper (some eager readers may go back to some sections of the paper). So, you want to introduce your results early on (another reason you may want to go back to the introduction multiple times). Highlight the results that are more interesting and link them to the context.

According to David Evans , some authors prefer to alternate between the introduction of one of the empirical strategies, to those results, and then they introduce another empirical strategy and the results. This strategy may be useful if different empirical methodologies are used.

5.2.1.6 6. Mechanisms, Robustness and Limitations (Optional Section)

If you have some ideas about what drives your results (the mechanisms involved), you may want to indicate that here. Some of the current critiques towards economics (and probably social sciences in general) has been the strong focus on establishing causation, with little regard to the context surrounding this (if you want to hear more, there is this thread from Dani Rodrick ). Agency matters and if the paper can say something about this (sometimes this goes beyond our research), you should indicate it in the introduction.

You may also want to briefly indicate how your results are valid after trying different specifications or sources of data (this is called Robustness checks). But you also want to be honest about the limitations of your research. But here, do not diminish the importance of your project. After you indicate the limitations, finish the paragraph restating the importance of your findings.

5.2.1.7 7. Value Added

A very important section in the introduction, these paragraphs help readers (and reviewers) to show why is your work important. What are the specific contributions of your paper?

This section is different from section 3 in that it points out the detailed additions you are making to the field with your research. Both sections can be connected if that fits your paper, but it is quite important that you keep the focus on the contributions of your paper, even if you discuss some literature connected to it, but always with the focus of showing what your paper adds. References (literature review) should come after in the paper.

5.2.1.8 8. Roadmap

A convention for the papers, this section needs to be kept short and outline the organization of the paper. To make it more useful, you can highlight some details that might be important in certain sections. But you want to keep this section succint (most readers skip this paragraph altogether).

5.2.2 In summary

The introduction of your paper will play a huge role in defining the future of your paper. Do not waste this opportunity and use it as well as your North Star guiding your path throughout the rest of the paper.

5.3 Context (Literature Review)

Do you need a literature review section?

5.4 Conclusion

• Request new password
• Create a new account

An Introduction to Qualitative Research

Student resources, part 1 (chapters 1 – 5): foundations of qualitative research.

• U.S. Locations
• UMGC Europe
• Learn Online
• Find Answers
• 855-655-8682
• Current Students

Online Guide to Writing and Research

The research process, explore more of umgc.

• Online Guide to Writing

Structuring the Research Paper

Formal research structure.

These are the primary purposes for formal research:

enter the discourse, or conversation, of other writers and scholars in your field

learn how others in your field use primary and secondary resources

find and understand raw data and information

For the formal academic research assignment, consider an organizational pattern typically used for primary academic research.  The pattern includes the following: introduction, methods, results, discussion, and conclusions/recommendations.

Usually, research papers flow from the general to the specific and back to the general in their organization. The introduction uses a general-to-specific movement in its organization, establishing the thesis and setting the context for the conversation. The methods and results sections are more detailed and specific, providing support for the generalizations made in the introduction. The discussion section moves toward an increasingly more general discussion of the subject, leading to the conclusions and recommendations, which then generalize the conversation again.

Sections of a Formal Structure

The introduction section.

Many students will find that writing a structured  introduction  gets them started and gives them the focus needed to significantly improve their entire paper. 

Introductions usually have three parts:

presentation of the problem statement, the topic, or the research inquiry

purpose and focus of your paper

summary or overview of the writer’s position or arguments

In the first part of the introduction—the presentation of the problem or the research inquiry—state the problem or express it so that the question is implied. Then, sketch the background on the problem and review the literature on it to give your readers a context that shows them how your research inquiry fits into the conversation currently ongoing in your subject area. 

In the second part of the introduction, state your purpose and focus. Here, you may even present your actual thesis. Sometimes your purpose statement can take the place of the thesis by letting your reader know your intentions. 

The third part of the introduction, the summary or overview of the paper, briefly leads readers through the discussion, forecasting the main ideas and giving readers a blueprint for the paper. 

The following example provides a blueprint for a well-organized introduction.

Example of an Introduction

Entrepreneurial Marketing: The Critical Difference

In an article in the Harvard Business Review, John A. Welsh and Jerry F. White remind us that “a small business is not a little big business.” An entrepreneur is not a multinational conglomerate but a profit-seeking individual. To survive, he must have a different outlook and must apply different principles to his endeavors than does the president of a large or even medium-sized corporation. Not only does the scale of small and big businesses differ, but small businesses also suffer from what the Harvard Business Review article calls “resource poverty.” This is a problem and opportunity that requires an entirely different approach to marketing. Where large ad budgets are not necessary or feasible, where expensive ad production squanders limited capital, where every marketing dollar must do the work of two dollars, if not five dollars or even ten, where a person’s company, capital, and material well-being are all on the line—that is, where guerrilla marketing can save the day and secure the bottom line (Levinson, 1984, p. 9).

By reviewing the introductions to research articles in the discipline in which you are writing your research paper, you can get an idea of what is considered the norm for that discipline. Study several of these before you begin your paper so that you know what may be expected. If you are unsure of the kind of introduction your paper needs, ask your professor for more information.  The introduction is normally written in present tense.

THE METHODS SECTION

The methods section of your research paper should describe in detail what methodology and special materials if any, you used to think through or perform your research. You should include any materials you used or designed for yourself, such as questionnaires or interview questions, to generate data or information for your research paper. You want to include any methodologies that are specific to your particular field of study, such as lab procedures for a lab experiment or data-gathering instruments for field research. The methods section is usually written in the past tense.

THE RESULTS SECTION

How you present the results of your research depends on what kind of research you did, your subject matter, and your readers’ expectations. 

Quantitative information —data that can be measured—can be presented systematically and economically in tables, charts, and graphs. Quantitative information includes quantities and comparisons of sets of data. 

Qualitative information , which includes brief descriptions, explanations, or instructions, can also be presented in prose tables. This kind of descriptive or explanatory information, however, is often presented in essay-like prose or even lists.

There are specific conventions for creating tables, charts, and graphs and organizing the information they contain. In general, you should use them only when you are sure they will enlighten your readers rather than confuse them. In the accompanying explanation and discussion, always refer to the graphic by number and explain specifically what you are referring to; you can also provide a caption for the graphic. The rule of thumb for presenting a graphic is first to introduce it by name, show it, and then interpret it. The results section is usually written in the past tense.

THE DISCUSSION SECTION

Your discussion section should generalize what you have learned from your research. One way to generalize is to explain the consequences or meaning of your results and then make your points that support and refer back to the statements you made in your introduction. Your discussion should be organized so that it relates directly to your thesis. You want to avoid introducing new ideas here or discussing tangential issues not directly related to the exploration and discovery of your thesis. The discussion section, along with the introduction, is usually written in the present tense.

THE CONCLUSIONS AND RECOMMENDATIONS SECTION

Your conclusion ties your research to your thesis, binding together all the main ideas in your thinking and writing. By presenting the logical outcome of your research and thinking, your conclusion answers your research inquiry for your reader. Your conclusions should relate directly to the ideas presented in your introduction section and should not present any new ideas.

You may be asked to present your recommendations separately in your research assignment. If so, you will want to add some elements to your conclusion section. For example, you may be asked to recommend a course of action, make a prediction, propose a solution to a problem, offer a judgment, or speculate on the implications and consequences of your ideas. The conclusions and recommendations section is usually written in the present tense.

Key Takeaways

• For the formal academic research assignment, consider an organizational pattern typically used for primary academic research. 
•  The pattern includes the following: introduction, methods, results, discussion, and conclusions/recommendations.

Mailing Address: 3501 University Blvd. East, Adelphi, MD 20783 This work is licensed under a  Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License . © 2022 UMGC. All links to external sites were verified at the time of publication. UMGC is not responsible for the validity or integrity of information located at external sites.

Table of Contents: Online Guide to Writing

Chapter 1: College Writing

How Does College Writing Differ from Workplace Writing?

What Is College Writing?

Why So Much Emphasis on Writing?

Chapter 2: The Writing Process

Doing Exploratory Research

Getting from Notes to Your Draft

Introduction

Prewriting - Techniques to Get Started - Mining Your Intuition

Prewriting: Targeting Your Audience

Prewriting: Techniques to Get Started

Prewriting: Understanding Your Assignment

Rewriting: Being Your Own Critic

Rewriting: Creating a Revision Strategy

Rewriting: Getting Feedback

Rewriting: The Final Draft

Techniques to Get Started - Outlining

Techniques to Get Started - Using Systematic Techniques

Thesis Statement and Controlling Idea

Writing: Getting from Notes to Your Draft - Freewriting

Writing: Getting from Notes to Your Draft - Summarizing Your Ideas

Writing: Outlining What You Will Write

Chapter 3: Thinking Strategies

A Word About Style, Voice, and Tone

A Word About Style, Voice, and Tone: Style Through Vocabulary and Diction

Critical Strategies and Writing

Critical Strategies and Writing: Analysis

Critical Strategies and Writing: Evaluation

Critical Strategies and Writing: Persuasion

Critical Strategies and Writing: Synthesis

Developing a Paper Using Strategies

Kinds of Assignments You Will Write

Patterns for Presenting Information

Patterns for Presenting Information: Critiques

Patterns for Presenting Information: Discussing Raw Data

Patterns for Presenting Information: General-to-Specific Pattern

Patterns for Presenting Information: Problem-Cause-Solution Pattern

Patterns for Presenting Information: Specific-to-General Pattern

Patterns for Presenting Information: Summaries and Abstracts

Supporting with Research and Examples

Writing Essay Examinations

Writing Essay Examinations: Make Your Answer Relevant and Complete

Writing Essay Examinations: Organize Thinking Before Writing

Writing Essay Examinations: Read and Understand the Question

Chapter 4: The Research Process

Planning and Writing a Research Paper

Planning and Writing a Research Paper: Ask a Research Question

Planning and Writing a Research Paper: Cite Sources

Planning and Writing a Research Paper: Collect Evidence

Planning and Writing a Research Paper: Decide Your Point of View, or Role, for Your Research

Planning and Writing a Research Paper: Draw Conclusions

Planning and Writing a Research Paper: Find a Topic and Get an Overview

Planning and Writing a Research Paper: Manage Your Resources

Planning and Writing a Research Paper: Outline

Planning and Writing a Research Paper: Survey the Literature

Planning and Writing a Research Paper: Work Your Sources into Your Research Writing

Research Resources: Where Are Research Resources Found? - Human Resources

Research Resources: What Are Research Resources?

Research Resources: Where Are Research Resources Found?

Research Resources: Where Are Research Resources Found? - Electronic Resources

Research Resources: Where Are Research Resources Found? - Print Resources

Structuring the Research Paper: Formal Research Structure

Structuring the Research Paper: Informal Research Structure

The Nature of Research

The Research Assignment: How Should Research Sources Be Evaluated?

The Research Assignment: When Is Research Needed?

The Research Assignment: Why Perform Research?

Chapter 5: Academic Integrity

Academic Integrity

Giving Credit to Sources

Giving Credit to Sources: Copyright Laws

Giving Credit to Sources: Documentation

Giving Credit to Sources: Style Guides

Integrating Sources

Practicing Academic Integrity

Practicing Academic Integrity: Keeping Accurate Records

Practicing Academic Integrity: Managing Source Material

Practicing Academic Integrity: Managing Source Material - Paraphrasing Your Source

Practicing Academic Integrity: Managing Source Material - Quoting Your Source

Practicing Academic Integrity: Managing Source Material - Summarizing Your Sources

Types of Documentation

Types of Documentation: Bibliographies and Source Lists

Types of Documentation: Citing World Wide Web Sources

Types of Documentation: In-Text or Parenthetical Citations

Types of Documentation: In-Text or Parenthetical Citations - APA Style

Types of Documentation: In-Text or Parenthetical Citations - CSE/CBE Style

Types of Documentation: In-Text or Parenthetical Citations - Chicago Style

Types of Documentation: In-Text or Parenthetical Citations - MLA Style

Types of Documentation: Note Citations

Chapter 6: Using Library Resources

Finding Library Resources

Chapter 7: Assessing Your Writing

How Is Writing Graded?

How Is Writing Graded?: A General Assessment Tool

The Draft Stage

The Draft Stage: The First Draft

The Draft Stage: The Revision Process and the Final Draft

The Draft Stage: Using Feedback

The Research Stage

Using Assessment to Improve Your Writing

Chapter 8: Other Frequently Assigned Papers

Reviews and Reaction Papers: Article and Book Reviews

Reviews and Reaction Papers: Reaction Papers

Writing Arguments

Writing Arguments: Adapting the Argument Structure

Writing Arguments: Purposes of Argument

Writing Arguments: References to Consult for Writing Arguments

Writing Arguments: Steps to Writing an Argument - Anticipate Active Opposition

Writing Arguments: Steps to Writing an Argument - Determine Your Organization

Writing Arguments: Steps to Writing an Argument - Develop Your Argument

Writing Arguments: Steps to Writing an Argument - Introduce Your Argument

Writing Arguments: Steps to Writing an Argument - State Your Thesis or Proposition

Writing Arguments: Steps to Writing an Argument - Write Your Conclusion

Writing Arguments: Types of Argument

Appendix A: Books to Help Improve Your Writing

Dictionaries

General Style Manuals

Researching on the Internet

Special Style Manuals

Writing Handbooks

Appendix B: Collaborative Writing and Peer Reviewing

Collaborative Writing: Assignments to Accompany the Group Project

Collaborative Writing: Informal Progress Report

Collaborative Writing: Issues to Resolve

Collaborative Writing: Methodology

Collaborative Writing: Peer Evaluation

Collaborative Writing: Tasks of Collaborative Writing Group Members

Collaborative Writing: Writing Plan

General Introduction

Peer Reviewing

Appendix C: Developing an Improvement Plan

Working with Your Instructor’s Comments and Grades

Appendix D: Writing Plan and Project Schedule

Devising a Writing Project Plan and Schedule

Reviewing Your Plan with Others

By using our website you agree to our use of cookies. Learn more about how we use cookies by reading our  Privacy Policy .

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

C.1.1 Formatting a Research Paper

Learning objectives.

• Identify the major components of a research paper written using American Psychological Association (APA) style.
• Apply general APA style and formatting conventions in a research paper.

In this chapter, you will learn how to use APA style , the documentation and formatting style followed by the American Psychological Association, as well as MLA style , from the Modern Language Association. There are a few major formatting styles used in academic texts, including AMA, Chicago, and Turabian:

• AMA (American Medical Association) for medicine, health, and biological sciences
• APA (American Psychological Association) for education, psychology, and the social sciences
• Chicago —a common style used in everyday publications like magazines, newspapers, and books
• MLA (Modern Language Association) for English, literature, arts, and humanities
• Turabian —another common style designed for its universal application across all subjects and disciplines

While all the formatting and citation styles have their own use and applications, in this chapter we focus our attention on the two styles you are most likely to use in your academic studies: APA and MLA.

If you find that the rules of proper source documentation are difficult to keep straight, you are not alone. Writing a good research paper is, in and of itself, a major intellectual challenge. Having to follow detailed citation and formatting guidelines as well may seem like just one more task to add to an already-too-long list of requirements.

Following these guidelines, however, serves several important purposes. First, it signals to your readers that your paper should be taken seriously as a student’s contribution to a given academic or professional field; it is the literary equivalent of wearing a tailored suit to a job interview. Second, it shows that you respect other people’s work enough to give them proper credit for it. Finally, it helps your reader find additional materials if he or she wishes to learn more about your topic.

Furthermore, producing a letter-perfect APA-style paper need not be burdensome. Yes, it requires careful attention to detail. However, you can simplify the process if you keep these broad guidelines in mind:

• Work ahead whenever you can. You can consult the chapter, “ Writing from Research: What Will I Learn? “ from the original version of this textbook ( intentionally omitted ), which  includes tips for keeping track of your sources early in the research process, which will save time later on.
• Get it right the first time. Apply APA guidelines as you write, so you will not have much to correct during the editing stage. Again, putting in a little extra time early on can save time later.
• Use the resources available to you. In addition to the guidelines provided in this chapter, you may wish to consult the APA website at http://www.apa.org or the Purdue University Online Writing lab at http://owl.english.purdue.edu , which regularly updates its online style guidelines.
• Consult the Fanshawe College Library website section “ ACADEMIC WRITING & CITATION ” for additional resources on research and citation.

General Formatting Guidelines

This chapter provides detailed guidelines for using the citation and formatting conventions developed by the American Psychological Association, or APA. Writers in disciplines as diverse as astrophysics, biology, psychology, and education follow APA style. The major components of a paper written in APA style are listed in the following box.

These are the major components of an APA-style paper:

Body, which includes the following:

• Headings and, if necessary, subheadings to organize the content
• In-text citations of research sources
• References page

All these components must be saved in one document, not as separate documents.

The title page of your paper includes the following information:

• Title of the paper
• Author’s name
• Name of the institution with which the author is affiliated
• Header at the top of the page with the paper title (in capital letters) and the page number (If the title is lengthy, you may use a shortened form of it in the header.)

List the first three elements in the order given in the previous list, centered about one third of the way down from the top of the page. Use the headers and footers tool of your word-processing program to add the header, with the title text at the left and the page number in the upper-right corner. Your title page should look like the following example.

The next page of your paper provides an abstract , or brief summary of your findings. An abstract does not need to be provided in every paper, but an abstract should be used in papers that include a hypothesis. A good abstract is concise—about one hundred fifty to two hundred fifty words—and is written in an objective, impersonal style. Your writing voice will not be as apparent here as in the body of your paper. When writing the abstract, take a just-the-facts approach, and summarize your research question and your findings in a few sentences.

Below is a sample paper written by a student named Jorge, who researched the effectiveness of low-carbohydrate diets. Read Jorge’s abstract. Note how it sums up the major ideas in his paper without going into excessive detail.

To view the original draft of this paper, you can review the chapter entitled,  “ Creating a Rough Draft for a Research Paper ” from the original version of this textbook ( intentionally omitted ).

Write an abstract summarizing your paper. Briefly introduce the topic, state your findings, and sum up what conclusions you can draw from your research. Use the word count feature of your word-processing program to make sure your abstract does not exceed one hundred fifty words.

Depending on your field of study, you may sometimes write research papers that present extensive primary research, such as your own experiment or survey. In your abstract, summarize your research question and your findings, and briefly indicate how your study relates to prior research in the field.

Margins, Pagination, and Headings

APA style requirements also address specific formatting concerns, such as margins, pagination, and heading styles, within the body of the paper. Review the following APA guidelines.

Use these general guidelines to format the paper:

• Set the top, bottom, and side margins of your paper at 1 inch.
• Use double-spaced text throughout your paper.
• Use a standard font, such as Times New Roman or Arial, in a legible size (10- to 12-point).
• Use continuous pagination throughout the paper, including the title page and the references section. Page numbers appear flush right within your header.
• Section headings and subsection headings within the body of your paper use different types of formatting depending on the level of information you are presenting. Additional details from Jorge’s paper are provided.

Begin formatting the final draft of your paper according to APA guidelines. You may work with an existing document or set up a new document if you choose. Include the following:

• Your title page
• The abstract you created in  “Exercise 1”
• Correct headers and page numbers for your title page and abstract

APA style uses  section headings  to organize information, making it easy for the reader to follow the writer’s train of thought and to know immediately what major topics are covered. Depending on the length and complexity of the paper, its major sections may also be divided into subsections, sub-subsections, and so on. These smaller sections, in turn, use different heading styles to indicate different levels of information. In essence, you are using headings to create a hierarchy of information.

The following heading styles used in APA formatting are listed in order of greatest to least importance:

• Section headings use centered, boldface type. Headings use title case, with important words in the heading capitalized.
• Subsection headings use left-aligned, boldface type. Headings use title case.
• The third level uses left-aligned, indented, boldface type. Headings use a capital letter only for the first word, and they end in a period.
• The fourth level follows the same style used for the previous level, but the headings are boldfaced and italicized.
• The fifth level follows the same style used for the previous level, but the headings are italicized and not boldfaced.

Visually, the hierarchy of information is organized as indicated in Table C.1.1 “Section Headings” .

Table C.1.1 Section Headings

A college research paper may not use all the heading levels shown in Table C.1.1 “Section Headings” , but you are likely to encounter them in academic journal articles that use APA style. For a brief paper, you may find that level 1 headings suffice. Longer or more complex papers may need level 2 headings or other lower-level headings to organize information clearly. Use your outline to craft your major section headings and determine whether any subtopics are substantial enough to require additional levels of headings.

Working with the document you developed in “Exercise 2” , begin setting up the heading structure of the final draft of your research paper according to APA guidelines. Include your title and at least two to three major section headings, and follow the formatting guidelines provided above. If your major sections should be broken into subsections, add those headings as well. Use your outline to help you.

Because Jorge used only level 1 headings, his Exercise 3 would look like the following:

Citation Guidelines

In-text citations.

Throughout the body of your paper, include a citation whenever you quote or paraphrase material from your research sources.  The purpose of citations is twofold: to give credit to others for their ideas and to allow your reader to follow up and learn more about the topic if desired. Your in-text citations provide basic information about your source; each source you cite will have a longer entry in the references section that provides more detailed information.

In-text citations must provide the name of the author or authors and the year the source was published. (When a given source does not list an individual author, you may provide the source title or the name of the organization that published the material instead.) When directly quoting a source, it is also required that you include the page number where the quote appears in your citation.

This information may be included within the sentence or in a parenthetical reference at the end of the sentence, as in these examples.

Epstein (2010) points out that “junk food cannot be considered addictive in the same way that we think of psychoactive drugs as addictive” (p. 137).

Here, the writer names the source author when introducing the quote and provides the publication date in parentheses after the author’s name. The page number appears in parentheses after the closing quotation marks and before the period that ends the sentence.

Addiction researchers caution that “junk food cannot be considered addictive in the same way that we think of psychoactive drugs as addictive” (Epstein, 2010, p. 137).

Here, the writer provides a parenthetical citation at the end of the sentence that includes the author’s name, the year of publication, and the page number separated by commas. Again, the parenthetical citation is placed after the closing quotation marks and before the period at the end of the sentence.

As noted in the book Junk Food, Junk Science (Epstein, 2010, p. 137), “junk food cannot be considered addictive in the same way that we think of psychoactive drugs as addictive.”

Here, the writer chose to mention the source title in the sentence (an optional piece of information to include) and followed the title with a parenthetical citation. Note that the parenthetical citation is placed before the comma that signals the end of the introductory phrase.

David Epstein’s book Junk Food, Junk Science (2010) pointed out that “junk food cannot be considered addictive in the same way that we think of psychoactive drugs as addictive” (p. 137).

Another variation is to introduce the author and the source title in your sentence and include the publication date and page number in parentheses within the sentence or at the end of the sentence. As long as you have included the essential information, you can choose the option that works best for that particular sentence and source.

Citing a book with a single author is usually a straightforward task. Of course, your research may require that you cite many other types of sources, such as books or articles with more than one author or sources with no individual author listed. You may also need to cite sources available in both print and online and nonprint sources, such as websites and personal interviews.

The rest of this chapter provides extensive guidelines for citing a variety of source types.

Writing at Work

APA is just one of several different styles with its own guidelines for documentation, formatting, and language usage. Depending on your field of interest, you may be exposed to additional styles, such as the following:

• MLA style. Determined by the Modern Languages Association and used for papers in literature, languages, and other disciplines in the humanities.
• Chicago style. Outlined in the Chicago Manual of Style and sometimes used for papers in the humanities and the sciences; many professional organizations use this style for publications as well.
• Associated Press (AP) style. Used by professional journalists.

References List

The brief citations included in the body of your paper correspond to the more detailed citations provided at the end of the paper in the references section. In-text citations provide basic information—the author’s name, the publication date, and the page number if necessary—while the references section provides more extensive bibliographical information. Again, this information allows your reader to follow up on the sources you cited and do additional reading about the topic if desired.

The specific format of entries in the list of references varies slightly for different source types, but the entries generally include the following information:

• The name(s) of the author(s) or institution that wrote the source
• The year of publication and, where applicable, the exact date of publication
• The full title of the source
• For books, the city of publication
• For articles or essays, the name of the periodical or book in which the article or essay appears
• For magazine and journal articles, the volume number, issue number, and pages where the article appears
• For sources on the web, the URL where the source is located

The references page is double spaced and lists entries in alphabetical order by the author’s last name. If an entry continues for more than one line, the second line and each subsequent line are indented five spaces. Review the following example.

In APA style, book and article titles are formatted in sentence case, not title case. Sentence case means that only the first word is capitalized, along with any proper nouns.

Key Takeaways

• Following proper citation and formatting guidelines helps writers ensure that their work will be taken seriously, give proper credit to other authors for their work, and provide valuable information to readers.
• Working ahead and taking care to cite sources correctly the first time are ways writers can save time during the editing stage of writing a research paper.
• APA papers usually include an abstract that concisely summarizes the paper.
• APA papers use a specific headings structure to provide a clear hierarchy of information.
• In APA papers, in-text citations usually include the name(s) of the author(s) and the year of publication.
• In-text citations correspond to entries in the references section, which provide detailed bibliographical information about a source.

Putting the Pieces Together Copyright © 2020 by Andrew Stracuzzi and André Cormier is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Research K to 12 chapters 1 to 5 READINESS

Related Papers

Janice Palmer

Analyze the data you have collected. What do you know about the students and what implications are there in the data? Do the students need small group instruction, more time practicing individually, or have they mastered the content and need to be challenged? A teacher must balance the administration of formative and summative assessments. The teacher does not want to misuse or over assess learners because they want to obtain true and valid data. According to Alkharusi, learners are identified by three achievement goals: 1. Mastery goals that focus on improving competence 2. Performance approach goals that focus on displaying competence 3. Performance avoidance goals that focus on avoiding a display of incompetence (Alkharusi, 2008) (pg. 244) Teachers want learners to be excited about learning and the display of what they have learned. This is why it is critical for teachers to synthesize assessment task for a valid picture of what skill learners have actually mastered. There are two types of assessments methods according to Alkharusi, the traditional and alternative method. The traditional method consists of true-false, multiple choice, and matching items which are low in complexities in terms of the skills being assessed. The traditional method does not require a lot of time to score or administer. Then there is the alternative method. The alternative method consists of performance-based assessments, portfolios, and observations and are high in complexities in terms of the skills being assessed. Alternative method requires more time to set up and administer as well as the time to score the results (Alkharusi, 2008) (pg. 247). Teachers are now required to implement the alternative methods because research has shown that alternative assessment methods are naturally motivating because students are asked to construct and or produce something compared to circling a letter in a multiple choice assessment or placing an X next to true or false. Knowing the achievement goals of learners will assist teachers in selecting different types of assessment activities to keep learners fully engaged and motivated. The three types of assessment that I will be using to conduct my comparison will be a problem based group project, a summative assessment of the unit, and the districts STARR assessment (Alkharusi, 2008) (pg. 247). Alkharusi, Aldhafri, Alnabhani, and Alkalbani (2012) " found, in their study of classroom assessment practices of 165 teachers in Oman, that teachers reported taking into account both effort and behavior when grading academic achievement. " Here at the Cook County Juvenile Detention Center (CCJDC) the behavior and the attitudes of the learners are very important to classroom management and student success. It is imperative that students in this environment be graded with the non-achievement grading methods. Although, over 50% of the students enrolled in our school has and Individual Learning Plan (IEP) or a Behavioral Plan (BP). Also, many have not attended school or attended school on a regular basis in over 6 months to 12 months +. While not many are at grade level the effort and time put into formative assessment and assessment activities are an integral part of the grading system. Teachers in this setting want to build self-esteem and advocate good classroom behavior something most if not all (in individual classrooms) of these students lack. So their perceptions of the assessment task can Type of Assessment Student A Student B

FRENGIE BOTOY

British Educational Research Journal

Simone Doolaard

isok franche

Psychology and Education: A Multidisciplinary Journal

Psychology and Education , Rodel A. Calvez

The study aimed to find out the reasons why the country's performance in domestic and international learning assessments is very low. It investigated the issues and concerns experienced by teachers in the workplace that affected the quality of their life and work-life situations, that further determines the quality of teachers in public elementary schools. The study took on concepts from Self-Determination Theory (SDT) to determine how teachers face the challenges in the system of Philippine education. It used open-ended questionnaires, where data were validated through personal interviews and focus group discussions (FGD). Results revealed that teachers have high level of resiliency, resourcefulness, creativity, and innovativeness in confronting challenges and are fully aware about the importance of personal well-being and of their family. The findings imply that education policies affect the quality of teachers to some extent making them enduring drivers of quality learning outcomes. The study recommends that education policies on the welfare of teachers be revisited and reviewed and be realigned to uphold teachers' mental health and general well-being.

Encyclopedia of the UN sustainable development goals

Eleonora Concina

Ehf- jhaye Raton

Yuchengco Center

RELATED PAPERS

Yolanda de maldonado

Autonomic Neuroscience

Norihiro Suzuki

Henry Briceño

Revista de la Asociación Colombiana de Dermatología y Cirugía Dermatológica

Leonardo Albis

The Linacre Quarterly

Walter Schumm

Dimitrios Anyfantis

Bhanwar Puniya

Biological Psychiatry

IEEE Transactions on Systems, Man, and Cybernetics: Systems

Rahib Abiyev

sebastien mosbah-natanson

REV Journal on Electronics and Communications

Tetrahedron Letters

Maria Camila Fernandez Silva

Castilla: Estudios de literatura

Emilio Suarez

Saúde (Santa Maria)

Taciane Barbosa

Jurnal Ilmiah Universitas Batanghari Jambi

Shaufi Arfintana

The Proceedings of Conference of Kyushu Branch

Bulletin of the American Physical Society

Yash Jaiswal

Journal of Business and Social Review in Emerging Economies

The Pakistan Development Review

Mohsin khan Mohsin khan

Mary Canellopoulou

Kozo Hiramatsu

wiyada kwanhian

Epilepsy Research

Danielle Ibarrola

Sabrina Celestino

Navina Jafa

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

This paper is in the following e-collection/theme issue:

Published on 19.4.2024 in Vol 26 (2024)

Psychometric Evaluation of a Tablet-Based Tool to Detect Mild Cognitive Impairment in Older Adults: Mixed Methods Study

Authors of this article:

Original Paper

• Josephine McMurray 1, 2 * , MBA, PhD   ; 
• AnneMarie Levy 1 * , MSc, PhD   ; 
• Wei Pang 1, 3 * , BTM   ; 
• Paul Holyoke 4 , PhD  

1 Lazaridis School of Business & Economics, Wilfrid Laurier University, Brantford, ON, Canada

2 Health Studies, Faculty of Human and Social Sciences, Wilfrid Laurier University, Brantford, ON, Canada

3 Biomedical Informatics & Data Science, Yale University, New Haven, CT, United States

4 SE Research Centre, Markham, ON, Canada

*these authors contributed equally

Corresponding Author:

Josephine McMurray, MBA, PhD

Lazaridis School of Business & Economics

Wilfrid Laurier University

73 George St

Brantford, ON, N3T3Y3

Phone: 1 548 889 4492

Email: [email protected]

Background: With the rapid aging of the global population, the prevalence of mild cognitive impairment (MCI) and dementia is anticipated to surge worldwide. MCI serves as an intermediary stage between normal aging and dementia, necessitating more sensitive and effective screening tools for early identification and intervention. The BrainFx SCREEN is a novel digital tool designed to assess cognitive impairment. This study evaluated its efficacy as a screening tool for MCI in primary care settings, particularly in the context of an aging population and the growing integration of digital health solutions.

Objective: The primary objective was to assess the validity, reliability, and applicability of the BrainFx SCREEN (hereafter, the SCREEN) for MCI screening in a primary care context. We conducted an exploratory study comparing the SCREEN with an established screening tool, the Quick Mild Cognitive Impairment (Qmci) screen.

Methods: A concurrent mixed methods, prospective study using a quasi-experimental design was conducted with 147 participants from 5 primary care Family Health Teams (FHTs; characterized by multidisciplinary practice and capitated funding) across southwestern Ontario, Canada. Participants included health care practitioners, patients, and FHT administrative executives. Individuals aged ≥55 years with no history of MCI or diagnosis of dementia rostered in a participating FHT were eligible to participate. Participants were screened using both the SCREEN and Qmci. The study also incorporated the Geriatric Anxiety Scale–10 to assess general anxiety levels at each cognitive screening. The SCREEN’s scoring was compared against that of the Qmci and the clinical judgment of health care professionals. Statistical analyses included sensitivity, specificity, internal consistency, and test-retest reliability assessments.

Results: The study found that the SCREEN’s longer administration time and complex scoring algorithm, which is proprietary and unavailable for independent analysis, presented challenges. Its internal consistency, indicated by a Cronbach α of 0.63, was below the acceptable threshold. The test-retest reliability also showed limitations, with moderate intraclass correlation coefficient (0.54) and inadequate κ (0.15) values. Sensitivity and specificity were consistent (63.25% and 74.07%, respectively) between cross-tabulation and discrepant analysis. In addition, the study faced limitations due to its demographic skew (96/147, 65.3% female, well-educated participants), the absence of a comprehensive gold standard for MCI diagnosis, and financial constraints limiting the inclusion of confirmatory neuropsychological testing.

Conclusions: The SCREEN, in its current form, does not meet the necessary criteria for an optimal MCI screening tool in primary care settings, primarily due to its longer administration time and lower reliability. As the number of digital health technologies increases and evolves, further testing and refinement of tools such as the SCREEN are essential to ensure their efficacy and reliability in real-world clinical settings. This study advocates for continued research in this rapidly advancing field to better serve the aging population.

International Registered Report Identifier (IRRID): RR2-10.2196/25520

Introduction

Mild cognitive impairment (MCI) is a syndrome characterized by a slight but noticeable and measurable deterioration in cognitive abilities, predominantly memory and thinking skills, that is greater than expected for an individual’s age and educational level [ 1 , 2 ]. The functional impairments associated with MCI are subtle and often impair instrumental activities of daily living (ADL). Instrumental ADL include everyday tasks such as managing finances, cooking, shopping, or taking regularly prescribed medications and are considered more complex than ADL such as bathing, dressing, and toileting [ 3 , 4 ]. In cases in which memory impairment is the primary indicator of the disease, MCI is classified as amnesic MCI and when significant impairment of non–memory-related cognitive domains such as visual-spatial or executive functioning is dominant, MCI is classified as nonamnesic [ 5 ].

Cognitive decline, more so than cancer and cardiovascular disease, poses a substantial threat to an individual’s ability to live independently or at home with family caregivers [ 6 ]. The Centers for Disease Control and Prevention reports that 1 in 8 adults aged ≥60 years experiences memory loss and confusion, with 35% reporting functional difficulties with basic ADL [ 7 ]. The American Academy of Neurology estimates that the prevalence of MCI ranges from 13.4% to 42% in people aged ≥65 years [ 8 ], and a 2023 meta-analysis that included 233 studies and 676,974 participants aged ≥50 years estimated that the overall global prevalence of MCI is 19.7% [ 9 ]. Once diagnosed, the prognosis for MCI is variable, whereby the impairment may be reversible; the rate of decline may plateau; or it may progressively worsen and, in some cases, may be a prodromal stage to dementia [ 10 - 12 ]. While estimates vary based on sample (community vs clinical), annual rates of conversion from MCI to dementia range from 5% to 24% [ 11 , 12 ], and those who present with multiple domains of cognitive impairment are at higher risk of conversion [ 5 ].

The risk of developing MCI rises with age, and while there are no drug treatments for MCI, nonpharmacologic interventions may improve cognitive function, alleviate the burden on caregivers, and potentially delay institutionalization should MCI progress to dementia [ 13 ]. To overcome the challenges of early diagnosis, which currently depends on self-detection, family observation, or health care provider (HCP) recognition of symptoms, screening high-risk groups for MCI or dementia is suggested as a solution [ 13 ]. However, the Canadian Task Force on Preventive Health Care recommends against screening adults aged ≥65 years due to a lack of meaningful evidence from randomized controlled trials and the high false-positive rate [ 14 - 16 ]. The main objective of a screening test is to reduce morbidity or mortality in at-risk populations through early detection and intervention, with the anticipated benefits outweighing potential harms. Using brief screening tools in primary care might improve MCI case detection, allowing patients and families to address reversible causes, make lifestyle changes, and access disease-modifying treatments [ 17 ].

There is no agreement among experts as to which tests or groups of tests are most predictive of MCI [ 16 ], and the gold standard approach uses a combination of positive results from neuropsychological assessments, laboratory tests, and neuroimaging to infer a diagnosis [ 8 , 18 ]. The clinical heterogeneity of MCI complicates its diagnosis because it influences not only memory and thinking abilities but also mood, behavior, emotional regulation, and sensorimotor abilities, and patients may present with any combination of symptoms with varying rates of onset and decline [ 4 , 8 ]. For this reason, a collaborative approach between general practitioners and specialists (eg, geriatricians and neurologists) is often required to be confident in the diagnosis of MCI [ 8 , 19 , 20 ].

In Canada, diagnosis often begins with screening for cognitive impairment followed by referral for additional testing; this process takes, on average, 5 months [ 20 ]. The current usual practice screening tools for MCI are the Mini-Mental State Examination (MMSE) [ 21 , 22 ] and the Montreal Cognitive Assessment (MoCA) 8.1 [ 3 ]. Both are paper-and-pencil screens administered in 10 to 15 minutes, scored out of 30, and validated as MCI screening tools across diverse clinical samples [ 23 , 24 ]. Universally, the MMSE is most often used to screen for MCI [ 20 , 25 ] and consists of 20 items that measure orientation, immediate and delayed recall, attention and calculation, visual-spatial skills, verbal fluency, and writing. The MoCA 8.1 was developed to improve on the MMSE’s ability to detect early signs of MCI, placing greater emphasis on evaluating executive function as well as language, memory, visual-spatial skills, abstraction, attention, concentration, and orientation across 30 items [ 24 , 26 ]. Scores of <24 on the MMSE or ≤25 on the MoCA 8.1 signal probable MCI [ 21 , 27 ]. Lower cutoff scores for both screens have been recommended to address evidence that they lack specificity to detect mild and early cases of MCI [ 4 , 28 - 31 ]. The clinical efficacy of both screens for tracking change in cognition over time is limited as they are also subject to practice effects with repeated administration [ 32 ].

Novel screening tools, including the Quick Mild Cognitive Impairment (Qmci) screen, have been developed with the goal of improving the accuracy of detecting MCI [ 33 , 34 ]. The Qmci is a sensitive and specific tool that differentiates normal cognition from MCI and dementia and is more accurate at differentiating MCI from controls than either the MoCA 8.1 (Qmci area under the curve=0.97 vs MoCA 8.1 area under the curve=0.92) [ 25 , 35 ] or the Short MMSE [ 33 , 36 ]. It also demonstrates high test-retest reliability (intraclass correlation coefficient [ICC]=0.88) [ 37 ] and is clinically useful as a rapid screen for MCI as the Qmci mean is 4.5 (SD 1.3) minutes versus 9.5 (SD 2.8) minutes for the MoCA 8.1 [ 25 ].

The COVID-19 pandemic and the necessary shift to virtual health care accelerated the use of digital assessment tools, including MCI screening tools such as the electronic MoCA 8.1 [ 38 , 39 ], and the increased use and adoption of technology (eg, smartphones and tablets) by older adults suggests that a lack of proficiency with technology may not be a barrier to the use of such assessment tools [ 40 , 41 ]. BrainFx is a for-profit firm that creates proprietary software designed to assess cognition and changes in neurofunction that may be caused by neurodegenerative diseases (eg, MCI or dementia), stroke, concussions, or mental illness using ecologically relevant tasks (eg, prioritizing daily schedules and route finding on a map) [ 42 ]. Their assessments are administered via a tablet and stylus. The BrainFx 360 performance assessment (referred to hereafter as the 360) is a 90-minute digitally administered test that was designed to assess cognitive, physical, and psychosocial areas of neurofunction across 26 cognitive domains using 49 tasks that are timed and scored [ 42 ]. The BrainFx SCREEN (referred to hereafter as the SCREEN) is a short digital version of the 360 that includes 7 of the cognitive domains included in the 360, is estimated to take approximately 10 to 15 minutes to complete, and was designed to screen for early detection of cognitive impairment [ 43 , 44 ]. Upon completion of any BrainFx assessment, the results of the 360 or SCREEN are added to the BrainFx Living Brain Bank (LBB), which is an electronic database that stores all completed 360 and SCREEN assessments and is maintained by BrainFx. An electronic report is generated by BrainFx comparing an individual’s results to those of others collected and stored in the LBB. Normative data from the LBB are used to evaluate and compare an individual’s results.

The 360 has been used in clinical settings to assess neurofunction among youth [ 45 ] and anecdotally in other rehabilitation settings (T Milner, personal communication, May 2018). To date, research on the 360 indicates that it has been validated in healthy young adults (mean age 22.9, SD 2.4 years) and that the overall test-retest reliability of the tool is high (ICC=0.85) [ 42 ]. However, only 2 of the 7 tasks selected to be included in the SCREEN produced reliability coefficients of >0.70 (visual-spatial and problem-solving abilities) [ 42 ]. Jones et al [ 43 ] explored the acceptability and perceived usability of the SCREEN with a small sample (N=21) of Canadian Armed Forces veterans living with posttraumatic stress disorder. A structural equation model based on the Unified Theory of Acceptance and Use of Technology suggested that behavioral intent to use the SCREEN was predicted by facilitating conditions such as guidance during the test and appropriate resources to complete the test [ 43 ]. However, the validity, reliability, and sensitivity of the SCREEN for detecting cognitive impairment have not been tested.

McMurray et al [ 44 ] designed a protocol to assess the validity, reliability, and sensitivity of the SCREEN for detecting early signs of MCI in asymptomatic adults aged ≥55 years in a primary care setting (5 Family Health Teams [FHTs]). The protocol also used a series of semistructured interviews and surveys guided by the fit between individuals, task, technology, and environment framework [ 46 ], a health-specific model derived from the Task-Technology Fit model by Goodhue and Thompson [ 47 ], to explore the SCREEN’s acceptability and use by HCPs and patients in primary care settings (manuscript in preparation). This study is a psychometric evaluation of the SCREEN’s validity, reliability, and sensitivity for detecting MCI in asymptomatic adults aged ≥55 years in primary care settings.

Study Location, Design, and Data Collection

This was a concurrent, mixed methods, prospective study using a quasi-experimental design. Participants were recruited from 5 primary care FHTs (characterized by multidisciplinary practice and capitated funding) across southwestern Ontario, Canada. FHTs that used a registered occupational therapist on staff were eligible to participate in the study, and participating FHTs received a nominal compensatory payment for the time the HCPs spent in training; collecting data for the study; administering the SCREEN, Qmci, and Geriatric Anxiety Scale–10 (GAS-10); and communicating with the research team. A multipronged recruitment approach was used [ 44 ]. A designated occupational therapist at each location was provided with training and equipment to recruit participants, administer assessment tools, and submit collected data to the research team.

The research protocol describing the methods of both the quantitative and qualitative arms of the study is published elsewhere [ 44 ].

Ethical Considerations

This study was approved by the Wilfrid Laurier University Research Ethics Board (ORE 5820) and was reviewed and approved by each FHT. Participants (HCPs, patients, and administrative executives) read and signed an information and informed consent package in advance of taking part in the study. We complied with recommendations for obtaining informed consent and conducting qualitative interviews with persons with dementia when recruiting patients who may be affected by neurocognitive diseases [ 48 - 50 ]. In addition, at the end of each SCREEN assessment, patients were required to provide their consent (electronic signature) to contribute their anonymized scores to the database of SCREEN results maintained by BrainFx. Upon enrolling in the study, participants were assigned a unique identification number that was used in place of their name on all study documentation to anonymize the data and preserve their confidentiality. A master list matching participant names with their unique identification number was stored in a password-protected file by the administering HCP and principal investigator on the research team. The FHTs received a nominal compensatory payment to account for their HCPs’ time spent administering the SCREEN, collecting data for the study, and communicating with the research team. However, the individual HCPs who volunteered to participate and the patient participants were not financially compensated for taking part in the study.

Participants

Patients who were rostered with the FHT, were aged ≥55 years, and had no history of MCI or dementia diagnoses to better capture the population at risk of early signs of cognitive impairment were eligible to participate [ 51 , 52 ]. It was necessary for the participants to be rostered with the FHTs to ensure that the HCPs could access their electronic medical record to confirm eligibility and record the testing sessions and results and to ensure that there was a responsible physician for referral if indicated. As the SCREEN is administered using a tablet, participants had to be able to read and think in English and discern color, have adequate hearing and vision to interact with the administering HCP, read 12-point font on the tablet, and have adequate hand and arm function to manipulate and hold the tablet. The exclusion criteria used in the study included colorblindness and any disability that might impair the individual’s ability to hold and interact with the tablet. Prospective participants were also excluded based on a diagnosis of conditions that may result in MCI or dementia-like symptoms, including major depression that required hospitalization, psychiatric disorders (eg, schizophrenia and bipolar disorder), psychopathology, epilepsy, substance use disorders, or sleep apnea (without the use of a continuous positive airway pressure machine) [ 52 ]. Patients were required to complete a minimum of 2 screening sessions spaced 3 months apart to participate in the study and, depending on when they enrolled to participate, could complete a maximum of 4 screening sessions over a year.

Data Collection Instruments

Gas-10 instrument.

A standardized protocol was used to collect demographic data, randomly administer the SCREEN and the Qmci (a validated screening tool for MCI), and administer the GAS-10 immediately before and after the completion of the first MCI screen at each visit [ 44 ]. This was to assess participants’ general anxiety as it related to screening for cognitive impairment at the time of the assessment, any change in subjective ratings after completion of the first MCI screen, and change in anxiety between appointments. The GAS-10 is a 10-item, self-report screen for anxiety in older adults [ 53 ] developed for rapid screening of anxiety in clinical settings (the GAS-10 is the short form of the full 30-item Geriatric Anxiety Scale [GAS]) [ 54 ]. While 3 subscales are identified, the GAS is reported to be a unidimensional scale that assesses general anxiety [ 55 , 56 ]. Validation of the GAS-10 suggests that it is optimal for assessing average to moderate levels of anxiety in older adults, with subscale scores that are highly and positively correlated with the GAS and high internal consistency [ 53 ]. Participants were asked to use a 4-point Likert scale (0= not at all , 1= sometimes , 2= most of the time , and 3= all of the time ) to rate how often they had experienced each symptom over the previous week, including on the day the test was administered [ 54 ]. The GAS-10 has a maximum score of 30, with higher scores indicating higher levels of anxiety [ 53 , 54 , 57 ].

HCPs completed the required training to become certified BrainFx SCREEN administrators before the start of the study. To this end, HCPs completed a web-based training program (developed and administered through the BrainFx website) that included 3 self-directed training modules. For the purpose of the study, they also participated in 1 half-day in-person training session conducted by a certified BrainFx administrator (T Milner, BrainFx chief executive officer) at one of the participating FHT locations. The SCREEN (version 0.5; beta) was administered on a tablet (ASUS ZenPad 10.1” IPS WXGA display, 1920 × 1200, powered by a quad-core 1.5 GHz, 64-bit MediaTek MTK 8163A processor with 2 GB RAM and 16-GB storage). The tablet came with a tablet stand for optional use and a dedicated stylus that is recommended for completion of a subset of questions. At the start of the study, HCPs were provided with identical tablets preloaded with the SCREEN software for use in the study. The 7 tasks on the SCREEN are summarized in Table 1 and were taken directly from the 360 based on a clustering and regression analysis of LBB records in 2016 (N=188) [ 58 ]. A detailed description of the study and SCREEN administration procedures was published by McMurray et al [ 44 ].

An activity score is generated for each of the 7 tasks on the SCREEN. It is computed based on a combination of the accuracy of the participant’s response and the processing speed (time in seconds) that it takes to complete the task. The relative contribution of accuracy and processing speed to the final activity score for each task is proprietary to BrainFx and unknown to the research team. The participant’s activity score is compared to the mean activity score for the same task at the time of testing in the LBB. The mean activity score from the LBB may be based on the global reference population (ie, all available SCREEN results in the LBB), or the administering HCP may select a specific reference population by filtering according to factors including but not limited to age, sex, or diagnosis. If the participant’s activity score is >1 SD below the LBB activity score mean for that task, it is labeled as an area of challenge . Each of the 7 tasks on the SCREEN are evaluated independently of each other, producing a report with 7 activity scores showing the participant’s score, the LBB mean score, and the SD. The report also provides an overall performance and processing speed score. The overall performance score is an average of all 7 activity scores; however, the way in which the overall processing speed score is generated remains proprietary to BrainFx and unknown to the research team. Both the overall performance and processing speed scores are similarly evaluated against the LBB and identified as an area of challenge using the criteria described previously. For the purpose of this study, participants’ mean activity scores on the SCREEN were compared to the results of people aged ≥55 years in the LBB.

The Qmci evaluated 6 cognitive domains: orientation (10 points), registration (5 points), clock drawing (15 points), delayed recall (20 points), verbal fluency (20 points), and logical memory (30 points) [ 59 ]. Administering HCPs scored the text manually, with each subtest’s points contributing to the overall score out of 100 points, and the cutoff score to distinguish normal cognition from MCI was ≤67/100 [ 60 ]. Cutoffs to account for age and education have been validated and are recommended as the Qmci is sensitive to these factors [ 60 ]. A 2019 meta-analysis of the diagnostic accuracy of MCI screening tools reported that the sensitivity and specificity of the Qmci for distinguishing MCI from normal cognition is similar to usual standard-of-care tools (eg, the MoCA, Addenbrooke Cognitive Examination–Revised, Consortium to Establish a Registry for Alzheimer’s Disease battery total score, and Sunderland Clock Drawing Test) [ 61 ]. The Qmci has also been translated into >15 different languages and has undergone psychometric evaluation across a subset of these languages. While not as broadly adopted as the MoCA 8.1 in Canada, its psychometric properties, administration time, and availability for use suggested that the Qmci was an optimal assessment tool for MCI screening in FHT settings during the study.

Psychometric Evaluation

To date, the only published psychometric evaluation of any BrainFx tool is by Searles et al [ 42 ] in Athletic Training & Sports Health Care ; it assessed the test-retest reliability of the 360 in 15 healthy adults between the ages of 20 and 25 years. This study evaluated the psychometric properties of the SCREEN and included a statistical analysis of the tool’s internal consistency, construct validity, test-retest reliability, and sensitivity and specificity. McMurray et al [ 44 ] provide a detailed description of the data collection procedures for administration of the SCREEN and Qmci completed by participants at each visit.

Validity Testing

Face validity was outside the scope of this study but was implied, and assumptions are reported in the Results section. Construct validity, whether the 7 activities that make up the SCREEN were representative of MCI, was assessed through comparison with a substantive body of literature in the domain and through principal component analysis using varimax rotation. Criterion validity measures how closely the SCREEN results corresponded to the results of the Qmci (used here as an “imperfect gold standard” for identifying MCI in older adults) [ 62 ]. A BrainFx representative hypothesized that the ecological validity of the SCREEN questions (ie, using tasks that reflect real-world activities to detect early signs of cognitive impairment) [ 63 ] makes it a more sensitive tool than other screens (T Milner, personal communication, May 2018) and allows HCPs to equate activity scores on the SCREEN with real-world functional abilities. Criterion validity was explored first using cross-tabulations to calculate the sensitivity and specificity of the SCREEN compared to those of the Qmci. Conventional screens such as the Qmci are scored by taking the sum of correct responses on the screen and a cutoff score derived from normative data to distinguish normal cognition from MCI. The SCREEN used a different method of scoring whereby each of the 7 tasks was scored and evaluated independently of each other and there were no recommended guidelines for distinguishing normal cognition from MCI based on the aggregate areas of challenge identified by the SCREEN. Therefore, to compare the sensitivity and specificity of the SCREEN against those of the Qmci, the results of both screens were coded into a binary format as 1=healthy and 2=unhealthy, where healthy denoted no areas of challenge identified through the SCREEN and a Qmci score of ≥67. Conversely, unhealthy denoted one or more areas of challenge identified through the SCREEN and a Qmci score of <67.

Criterion validity was further explored using discrepant analysis via a resolver test [ 44 ]. Following the administration of the SCREEN and Qmci, screen results were evaluated by the administering HCP. HCPs were instructed to refer the participant for follow-up with their primary care physician if the Qmci result was <67 regardless of whether any areas of challenge were identified on the SCREEN. However, HCPs could use their clinical judgment to refer a participant for physician follow-up based on the results of the SCREEN or the Qmci, and all the referral decisions were charted on the participant’s electronic medical record following each visit and screening. In discrepant analysis, the results of the imperfect gold standard [ 64 ], as was the role of the Qmci in this study, were compared with the SCREEN results. A resolver test (classified as whether the HCP referred the patient to a physician for follow-up based on their performance on the SCREEN and the Qmci) was used on discordant results [ 64 , 65 ] to determine sensitivity and specificity. To this end, a new variable, Referral to a Physician for Cognitive Impairment , was coded as the true status (1=no referral; 2=referral was made) and compared to the Qmci as the imperfect gold standard (1=healthy; 2=unhealthy).

Reliability Testing

The reliability of a screening instrument is its ability to consistently measure an attribute and how well its component measures fit together conceptually. Internal consistency identifies whether the items in a multi-item scale are measuring the same underlying construct; the internal consistency of the SCREEN was assessed using the Cronbach α. Test-retest reliability refers to the ability of a measurement instrument to reproduce results over ≥2 occasions (assuming the underlying conditions have not changed) and was assessed using paired t tests (2-tailed), ICC, and the κ coefficient. In this study, participants completed both the SCREEN and the Qmci in the same sitting in a random sequence on at least 2 different occasions spaced 3 months apart (administration procedures are described elsewhere) [ 44 ]. In some instances, the screens were administered to the same participant on 4 separate occasions spaced 3 months apart each, and this provided up to 3 separate opportunities to conduct test-retest reliability analyses and investigate the effects of repeated practice. There are no clear guidelines on the optimal time between tests [ 66 , 67 ]; however, Streiner and Kottner [ 68 ] and Streiner [ 69 ] recommend longer periods between tests (eg, at least 10-14 days) to avoid recall bias, and greater practice effects have been experienced with shorter test-retest intervals [ 32 ].

Analysis of the quantitative data was completed using Stata (version 17.0; StataCorp). Assumptions of normality were not violated, so parametric tests were used. Collected data were reported using frequencies and percentages and compared using the chi-square or Fisher exact test as necessary. Continuous data were analyzed for central tendency and variability; categoric data were presented as proportions. Normality was tested using the Shapiro-Wilk test, and nonparametric data were tested using the Mann-Whitney U test. A P value of .05 was considered statistically significant, with 95% CIs provided where appropriate. We powered the exploratory analysis to validate the SCREEN using an estimated effect size of 12%—understanding that Canadian prevalence rates of MCI were not available [ 1 ]—and determined that the study required at least 162 participants. For test-retest reliability, using 90% power and a 5% type-I error rate, a minimum of 67 test results was required.

The time taken for participants to complete the SCREEN was recorded by the HCPs at the time of testing; there were 6 missing HCP records of time to complete the SCREEN. For these 6 cases of missing data, we imputed the mean time to complete the SCREEN by all participants who were tested by that HCP and used this to populate the missing cells [ 70 ]. There were 3 cases of missing data related to the SCREEN reports. More specifically, the SCREEN report generated by BrainFx did not include 1 or 2 data points each for the route finding, divided attention, and prioritizing tasks. The clinical notes provided by the HCP at the time of SCREEN administration did not indicate that the participant had not completed those questions, and it was not possible to determine the root cause of the missing data in report generation according to BrainFx (M Milner, personal communication, July 7, 2020). For continuous variables in analyses such as exploratory factor analysis, Cronbach α, and t test, missing values were imputed using the mean. However, for the coded healthy and unhealthy categorical variables, values were not imputed.

Data collection began in January 2019 and was to conclude on May 31, 2020. However, the emergence of the global COVID-19 pandemic resulted in the FHTs and Wilfrid Laurier University prohibiting all in-person research starting on March 16, 2020.

Participant Demographics

A total of 154 participants were recruited for the study, and 20 (13%) withdrew following their first visit to the FHT. The data of 65% (13/20) of the participants who withdrew were included in the final analysis, and the data of the remaining 35% (7/20) were removed, either due to their explicit request (3/7, 43%) or because technical issues at the time of testing rendered their data unusable (4/7, 57%). These technical issues were related to software issues (eg, any instance in which the patient or HCP interacted with the SCREEN software and followed the instructions provided, the software did not work as expected [ie, objects did not move where they were dragged or tapping on objects failed to highlight the object], and the question could not be completed). After attrition, a total of 147 individuals aged ≥55 years with no previous diagnosis of MCI or dementia participated in the study ( Table 2 ). Of the 147 participants, 71 (48.3%) took part in only 1 round of screening on visit 1 (due to COVID-19 restrictions imposed on in-person research that prevented a second visit). The remaining 51.7% (76/147) of the participants took part in ≥2 rounds of screening across multiple visits (76/147, 51.7% participated in 2 rounds; 22/147, 15% participated in 3 rounds; and 13/147, 8.8% participated in 4 rounds of screening).

The sample population was 65.3% (96/147) female (mean 70.2, SD 7.9 years) and 34.7% (51/147) male (mean 72.5, SD 8.1 years), with age ranging from 55 to 88 years; 65.3% (96/147) achieved the equivalent of or higher than a college diploma or certificate ( Table 2 ); and 32.7% (48/147) self-reported living with one or more chronic medical conditions ( Table 3 ). At the time of screening, 73.5% (108/147) of participants were also taking medications with side effects that may include impairments to memory and thinking abilities [ 71 - 75 ]; therefore, medication use was accounted for in a subset of the analyses. Finally, 84.4% (124/147) of participants self-reported regularly using technology (eg, smartphone, laptop, or tablet) with high proficiency. A random sequence generator was used to determine the order for administering the MCI screens; the SCREEN was administered first 51.9% (134/258) of the time.

Construct Validity

Construct validity was assessed through a review of relevant peer-reviewed literature that compared constructs included in the SCREEN with those identified in the literature as 2 of the most sensitive tools for MCI screening: the MoCA 8.1 [ 76 ] and the Qmci [ 25 ]. Memory, language, and verbal skills are assessed in the MoCA and Qmci but are absent from the SCREEN. Tests of verbal fluency and logical memory have been shown to be particularly sensitive to early cognitive changes [ 77 , 78 ] but are similarly absent from the SCREEN.

Exploratory factor analysis was performed to examine the SCREEN’s ability to reliably measure risk of MCI. The Kaiser-Meyer-Olkin measure yielded a value of 0.79, exceeding the commonly accepted threshold of 0.70, indicating that the sample was adequate for factor analysis. The Bartlett test of sphericity returned a chi-square value of χ 2 21 =167.1 ( P <.001), confirming the presence of correlations among variables suitable for factor analysis. A principal component analysis revealed 2 components with eigenvalues of >1, cumulatively accounting for 52.12% of the variance, with the first factor alone explaining 37.8%. After the varimax rotation, the 2 factors exhibited distinct patterns of loadings, with the visual-spatial ability factor loading predominantly on the second factor. The SCREEN tasks, except for visual-spatial ability, loaded substantially on the factors (>0.5), suggesting that the SCREEN possesses good convergent validity for assessing the risk of MCI.

Criterion Validity

The coding of SCREEN scores into a binary healthy and unhealthy outcome standardized the dependent variable to allow for criterion testing. Criterion validity was assessed using cross-tabulations and the analysis of confusion matrices and provided insights into the sensitivity and specificity of the SCREEN when compared to the Qmci. Of the 144 cases considered, 20 (13.9%) were true negatives, and 74 (51.4%) were true positives. The SCREEN’s sensitivity, which reflects its capacity to accurately identify healthy individuals (true positives), was 63.25% (74 correct identifications/117 actual positives). The specificity of the test, indicating its ability to accurately identify unhealthy individuals (true negatives), was 74.07% (20 correct identifications/27 actual negatives). Then, sensitivity and specificity were derived using discrepant analysis and a resolver test previously described (whether the HCP referred the participant to a physician following the screens). The results were identical, the estimate of the SCREEN sensitivity was 63.3% (74/117), and the estimate of the specificity was 74% (20/27).

Internal Reliability

A Cronbach α=0.70 is acceptable, and at least 0.90 is required for clinical instruments [ 79 ]. The estimate of internal consistency for the SCREEN (N=147) was Cronbach α=0.63.

Test-Retest Reliability

Test-retest reliability analyses were conducted using ICC for the SCREEN activity scores and the κ coefficient for the healthy and unhealthy classifications. Guidelines for interpretation of the ICC suggest that anything <0.5 indicates poor reliability and anything between 0.5 and 0.75 suggests moderate reliability [ 80 ]; the ICC for the SCREEN activity scores was 0.54. With respect to the κ coefficient, a κ value of <0.2 is considered to have no level of agreement, a κ value of 0.21 to 0.39 is considered minimal, a κ value of 0.4 to 0.59 is considered weak agreement, and anything >0.8 suggests strong to almost perfect agreement [ 81 ]. The κ coefficient for healthy and unhealthy classifications was 0.15.

Analysis of the Factors Impacting Healthy and Unhealthy Results

The Spearman rank correlation was used to assess the relationships between participants’ overall activity score on the SCREEN and their total time to complete the SCREEN; age, sex, and self-reported levels of education; technology use; medication use; amount of sleep; and level of anxiety (as measured using the GAS-10) at the time of SCREEN administration. Lower overall activity scores were moderately correlated with being older ( r s142 =–0.57; P <.001) and increased total time to complete the SCREEN ( r s142 =0.49; P <.001). There was also a moderate inverse relationship between overall activity score and total time to compete the SCREEN ( r s142 =–0.67; P <.001) whereby better performance was associated with quicker task completion. There were weak positive associations between overall activity score and increased technology use ( r s142 =0.34; P <.001) and higher level of education ( r s142 =0.21; P =.01).

A logistic regression model was used to predict the SCREEN result using data from 144 observations. The model’s predictors explain approximately 21.33% of the variance in the outcome variable. The likelihood ratio test indicates that the model provides a significantly better fit to the data than a model without predictors ( P <.001).

The SCREEN outcome variable ( healthy vs unhealthy ) was associated with the predictor variables sex and total time to complete the SCREEN. More specifically, female participants were more likely to obtain healthy SCREEN outcomes ( P =.007; 95% CI 0.32-2.05). For all participants, the longer it took to complete the SCREEN, the less likely they were to achieve a healthy SCREEN outcome ( P =.002; 95% CI –0.33 to –0.07). Age ( P =.25; 95% CI –0.09 to 0.02), medication use ( P =.96; 95% CI –0.9 to 0.94), technology use ( P =.44; 95% CI –0.28 to 0.65), level of education ( P =.14; 95% CI –0.09 to 0.64), level of anxiety ( P =.26; 95% CI –1.13 to 0.3), and hours of sleep ( P =.08; 95% CI –0.06 to 0.93) were not significant.

Impact of Practice Effects

The SCREEN was administered approximately 3 months apart, and separate, paired-sample t tests were performed to compare SCREEN outcomes between visits 1 and 2 (76/147, 51.7%; Table 4 ), visits 2 and 3 (22/147, 15%), and visits 3 and 4 (13/147, 8.8%). Declining visits were partially attributable to the early shutdown of data collection due to the COVID-19 pandemic, and therefore, comparisons between visits 2 and 3 or visits 3 and 4 were not reported. Compared to participants’ SCREEN performance on visit 1, their overall mean activity score and overall processing time improved on their second administration of the SCREEN (score: t 75 =–2.86 and P =.005; processing time: t 75 =–2.98 and P =.004). Even though the 7 task-specific activity scores on the SCREEN also increased between visits 1 and 2, these improvements were not significant, indicating that the difference in overall activity scores was cumulative and not attributable to a specific task ( Table 4 ).

Principal Findings

Our study aimed to evaluate the effectiveness and reliability of the BrainFx SCREEN in detecting MCI in primary care settings. The research took place during the COVID-19 pandemic, which influenced the study’s execution and timeline. Despite these challenges, the findings offer valuable insights into cognitive impairment screening.

Brief MCI screening tools help time-strapped primary care physicians determine whether referral for a definitive battery of more time-consuming and expensive tests is warranted. These tools must optimize and balance the need for time efficiency while also being psychometrically valid and easily administered [ 82 ]. The importance of brevity is determined by a number of factors, including the clinical setting. Screens that can be completed in approximately ≤5 minutes [ 13 ] are recommended for faster-paced clinical settings (eg, emergency rooms and preoperative screens), whereas those that can be completed in 5 to 10 minutes or less are better suited to primary care settings [ 82 - 84 ]. Identifying affordable, psychometrically tested screening tests for MCI that integrate into clinical workflows and are easy to consistently administer and complete may help with the following:

• Initiating appropriate diagnostic tests for signs and symptoms at an earlier stage
• Normalizing and destigmatizing cognitive testing for older adults
• Expediting referrals
• Allowing for timely access to programs and services that can support aging in place or delay institutionalization
• Reducing risk
• Improving the psychosocial well-being of patients and their care partners by increasing access to information and resources that aid with future planning and decision-making [ 85 , 86 ]

Various cognitive tests are commonly used for detecting MCI. These include the Addenbrook Cognitive Examination–Revised, Consortium to Establish a Registry for Alzheimer’s Disease, Sunderland Clock Drawing Test, Informant Questionnaire on Cognitive Decline in the Elderly, Memory Alternation Test, MMSE, MoCA 8.1, and Qmci [ 61 , 87 ]. The Addenbrook Cognitive Examination–Revised, Consortium to Establish a Registry for Alzheimer’s Disease, MoCA 8.1, Qmci, and Memory Alternation Test are reported to have similar diagnostic accuracy [ 61 , 88 ]. The HCPs participating in this study reported using the MoCA 8.1 as their primary screening tool for MCI along with other assessments such as the MMSE and Trail Making Test parts A and B.

Recent research highlights the growing use of digital tools [ 51 , 89 , 90 ], mobile technology [ 91 , 92 ], virtual reality [ 93 , 94 ], and artificial intelligence [ 95 ] to improve early identification of MCI. Demeyere et al [ 51 ] developed the tablet-based, 10-item Oxford Cognitive Screen–Plus to detect slight changes in cognitive impairment across 5 domains of cognition (memory, attention, number, praxis, and language), which has been validated among neurologically healthy older adults. Statsenko et al [ 96 ] have explored improvement of the predictive capabilities of tests using artificial intelligence. Similarly, there is an emerging focus on the use of machine learning techniques to detect dementia leveraging routinely collected clinical data [ 97 , 98 ]. This progression signifies a shift toward more technologically advanced, efficient, and potentially more accurate diagnostic approaches in the detection of MCI.

Whatever the modality, screening tools should be quick to administer, demonstrate consistent results over time and between different evaluators, cover all major cognitive areas, and be straightforward to both administer and interpret [ 99 ]. However, highly sensitive tests such as those suggested for screening carry a significant risk of false-positive diagnoses [ 15 ]. Given the high potential for harm of false positives, it is important to validate the psychometric properties of screening tests across different populations and understand how factors such as age and education can influence the results [ 99 ].

Our study did not assess the face validity of the SCREEN, but participating occupational therapists were comfortable with the test regimen. Nonetheless, the research team noted the absence of verbal fluency and memory tests in the SCREEN, both of which McDonnell et al [ 100 ] identified as being more sensitive to the more commonly seen amnesic MCI. Two of the most sensitive tools for MCI screening, the MoCA 8.1 [ 76 ] and Qmci [ 25 ], assess memory, language, and verbal skills, and tests of verbal fluency and logical memory have been shown to be particularly sensitive to early cognitive changes [ 77 , 78 ].

The constructs included in the SCREEN ( Table 1 ) were selected based on a single non–peer-reviewed study [ 58 ] using the 360 and traumatic brain injury data (N=188) that identified the constructs as predictive of brain injury. The absence of tasks that measure verbal fluency or logical memory in the SCREEN appears to weaken claims of construct validity. The principal component analysis of the SCREEN assessment identified 2 components accounting for 52.12% of the total variance. The first component was strongly associated with abstract reasoning, constructive ability, and divided attention, whereas the second was primarily influenced by visual-spatial abilities. This indicates that constructs related to perception, attention, and memory are central to the SCREEN scores.

The SCREEN’s binary outcome (healthy or unhealthy) created by the research team was based on comparisons with the Qmci. However, the method of identifying areas of challenge in the SCREEN by comparing the individual’s mean score on each of the 7 tasks with the mean scores of a global or filtered cohort in the LBB introduces potential biases or errors. These could arise from a surge in additions to the LBB from patients with specific characteristics, self-selection of participants, poorly trained SCREEN administrators, inclusion of nonstandard test results, underuse of appropriate filters, and underreporting of clinical conditions or factors such as socioeconomic status that impact performance in standardized cognitive tests.

The proprietary method of analyzing and reporting SCREEN results complicates traditional sensitivity and specificity measurement. Our testing indicated a sensitivity of 63.25% and specificity of 74.07% for identifying healthy (those without MCI) and unhealthy (those with MCI) individuals. The SCREEN’s Cronbach α=.63, slightly below the threshold for clinical instruments, and reliability scores that were lower than the ideal standards suggest a higher-than-acceptable level of random measurement error in its constructs. The lower reliability may also stem from an inadequate sample size or a limited number of scale items.

The SCREEN’s results are less favorable compared to those of other digital MCI screening tools that similarly enable evaluation of specific cognitive domains but also provide validated, norm-referenced cutoff scores and methods for cumulative scoring in clinical settings (Oxford Cognitive Screen–Plus) [ 51 ] or of validated MCI screening tools used in primary care (eg, MoCA 8.1, Qmci, and MMSE) [ 51 , 87 ]. The SCREEN’s unique scoring algorithm and the dynamic denominator in data analysis necessitate caution in comparing these results to those of other tools with fixed scoring algorithms and known sensitivities [ 101 , 102 ]. We found the SCREEN to have lower-than-expected internal reliability, suggesting significant random measurement error. Test-retest reliability was weak for the healthy or unhealthy outcome but stronger for overall activity scores between tests. The variability in identifying areas of challenge could relate to technological difficulties or variability from comparisons with a growing database of test results.

Potential reasons for older adults’ poorer scores on timed tests include the impact of sensorimotor decline on touch screen sensation and reaction time [ 38 , 103 ], anxiety related to taking a computer-enabled test [ 104 - 106 ], or the anticipated consequences of a negative outcome [ 107 ]. However, these effects were unlikely to have influenced the results of this study. Practice effects were observed [ 29 , 108 ], but the SCREEN’s novelty suggests that familiarity is not gained through prepreparation or word of mouth as this sample was self-selected and not randomized. Future research might also explore the impact of digital literacy and cultural differences in the interpretation of software constructs or icons on MCI screening in a randomized, older adult sample.

Limitations

This study had methodological limitations that warrant attention. The small sample size and the demographic distribution of the 147 participants aged ≥55 years, with most (96/147, 65.3%) being female and well educated, limits the generalizability of the findings to different populations. The study’s design, aiming to explore the sensitivity of the SCREEN for early detection of MCI, necessitated the exclusion of individuals with a previous diagnosis of MCI or dementia. This exclusion criterion might have impacted the study’s ability to thoroughly assess the SCREEN’s effectiveness in a more varied clinical context. The requirement for participants to read and comprehend English introduced another limitation to our study. This criterion potentially limited the SCREEN tool’s applicability across diverse linguistic backgrounds as individuals with language-based impairments or those not proficient in English may face challenges in completing the assessment [ 51 ]. Such limitations could impact the generalizability of our findings to non–English-speaking populations or to those with language impairments, underscoring the need for further research to evaluate the SCREEN tool’s effectiveness in broader clinical and linguistic contexts.

Financial constraints played a role in limiting the study’s scope. Due to funding limitations, it was not possible to include specialist assessments and a battery of neuropsychiatric tests generally considered the gold standard to confirm or rule out an MCI diagnosis. Therefore, the study relied on differential verification through 2 imperfect reference standards: a comparison with the Qmci (the tool with the highest published sensitivity to MCI in 2019, when the study was designed) and the clinical judgment of the administering HCP, particularly in decisions regarding referrals for further clinical assessment. Furthermore, while an economic feasibility assessment was considered, the research team determined that it should follow, not precede, an evaluation of the SCREEN’s validity and reliability.

The proprietary nature of the algorithm used for scoring the SCREEN posed another challenge. Without access to this algorithm, the research team had to use a novel comparative statistical approach, coding patient results into a binary variable: healthy (SCREEN=no areas of challenge OR Qmci≥67 out of 100) or unhealthy (SCREEN=one or more areas of challenge OR Qmci<67 out of 100). This may have introduced a higher level of error into our statistical analysis. Furthermore, the process for determining areas of challenge on the SCREEN involves comparing a participant’s result to the existing SCREEN results in the LBB at the time of testing. By the end of this study, the LBB contained 632 SCREEN results for adults aged ≥55 years, with this study contributing 258 of those results. The remaining 366 original SCREEN results, 64% of which were completed by individuals who self-identified as having a preexisting diagnosis or conditions associated with cognitive impairment (eg, traumatic brain injury, concussion, or stroke), could have led to an overestimation of the means and SDs of the study participants’ results at the outset of the study.

Unlike other cognitive screening tools, the SCREEN allows for filtering of results to compare different patient cohorts in the LBB using criteria such as age and education. However, at this stage of the LBB’s development, using such filters can significantly reduce the reliability of the results due to a smaller comparator population (ie, the denominator used to calculate the mean and SD). This, in turn, affects the significance of the results. Moreover, the constantly changing LBB data set makes it challenging to meaningfully compare an individual’s results over time as the evolving denominator affects the accuracy and relevance of these comparisons. Finally, the significant improvement in SCREEN scores between the first and second visits suggests the presence of practice effects, which could have influenced the reliability and validity of the findings.

Conclusions

In a primary care setting, where MCI screening tools are essential and recommended for those with concerns [ 85 ], certain criteria are paramount: time efficiency, ease of administration, and robust psychometric properties [ 82 ]. Our analysis of the BrainFx SCREEN suggests that, despite its innovative approach and digital delivery, it currently falls short in meeting these criteria. The SCREEN’s comparatively longer administration time and lower-than-expected reliability scores suggest that it may not be the most effective tool for MCI screening of older adults in a primary care setting at this time.

It is important to note that, in the wake of the COVID-19 pandemic, and with an aging population living and aging by design or necessity in a community setting, there is growing interest in digital solutions, including web-based applications and platforms to both collect digital biomarkers and deliver cognitive training and other interventions [ 109 , 110 ]. However, new normative standards are required when adapting cognitive tests to digital formats [ 92 ] as the change in medium can significantly impact test performance and results interpretation. Therefore, we recommend caution when interpreting our study results and encourage continued research and refinement of tools such as the SCREEN. This ongoing process will ensure that current and future MCI screening tools are effective, reliable, and relevant in meeting the needs of our aging population, particularly in primary care settings where early detection and intervention are key.

Acknowledgments

The researchers gratefully acknowledge the Ontario Centres of Excellence Health Technologies Fund for their financial support of this study; the executive directors and clinical leads in each of the Family Health Team study locations; the participants and their friends and families who took part in the study; and research assistants Sharmin Sharker, Kelly Zhu, and Muhammad Umair for their contributions to data management and statistical analysis.

Data Availability

The data sets generated during and analyzed during this study are available from the corresponding author on reasonable request.

Authors' Contributions

JM contributed to the conceptualization, methodology, validation, formal analysis, data curation, writing—original draft, writing—review and editing, visualization, supervision, and funding acquisition. AML contributed to the conceptualization, methodology, validation, investigation, formal analysis, data curation, writing—original draft, writing—review and editing, visualization, and project administration. WP contributed to the validation, formal analysis, data curation, writing—original draft, writing—review and editing, and visualization. Finally, PH contributed to conceptualization, methodology, writing—review and editing, supervision, and funding acquisition.

Conflicts of Interest

None declared.

• Casagrande M, Marselli G, Agostini F, Forte G, Favieri F, Guarino A. The complex burden of determining prevalence rates of mild cognitive impairment: a systematic review. Front Psychiatry. 2022;13:960648. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Petersen RC, Caracciolo B, Brayne C, Gauthier S, Jelic V, Fratiglioni L. Mild cognitive impairment: a concept in evolution. J Intern Med. Mar 2014;275(3):214-228. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Knopman DS, Petersen RC. Mild cognitive impairment and mild dementia: a clinical perspective. Mayo Clin Proc. Oct 2014;89(10):1452-1459. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Anderson ND. State of the science on mild cognitive impairment (MCI). CNS Spectr. Feb 2019;24(1):78-87. [ CrossRef ] [ Medline ]
• Tangalos EG, Petersen RC. Mild cognitive impairment in geriatrics. Clin Geriatr Med. Nov 2018;34(4):563-589. [ CrossRef ] [ Medline ]
• Ng R, Maxwell C, Yates E, Nylen K, Antflick J, Jette N, et al. Brain disorders in Ontario: prevalence, incidence and costs from health administrative data. Institute for Clinical Evaluative Sciences. 2015. URL: https:/​/www.​ices.on.ca/​publications/​research-reports/​brain-disorders-in-ontario-prevalence-incidence-and-costs-from-health-administrative-data/​ [accessed 2024-04-01]
• Centers for Disease ControlPrevention (CDC). Self-reported increased confusion or memory loss and associated functional difficulties among adults aged ≥ 60 years - 21 states, 2011. MMWR Morb Mortal Wkly Rep. May 10, 2013;62(18):347-350. [ FREE Full text ] [ Medline ]
• Petersen RC, Lopez O, Armstrong MJ, Getchius TS, Ganguli M, Gloss D, et al. Practice guideline update summary: mild cognitive impairment: report of the guideline development, dissemination, and implementation subcommittee of the American Academy of Neurology. Neurology. Jan 16, 2018;90(3):126-135. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Song WX, Wu WW, Zhao YY, Xu HL, Chen GC, Jin SY, et al. Evidence from a meta-analysis and systematic review reveals the global prevalence of mild cognitive impairment. Front Aging Neurosci. 2023;15:1227112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chen Y, Denny KG, Harvey D, Farias ST, Mungas D, DeCarli C, et al. Progression from normal cognition to mild cognitive impairment in a diverse clinic-based and community-based elderly cohort. Alzheimers Dement. Apr 2017;13(4):399-405. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Langa KM, Levine DA. The diagnosis and management of mild cognitive impairment: a clinical review. JAMA. Dec 17, 2014;312(23):2551-2561. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang Y, Natale G, Clouston S. Incidence of mild cognitive impairment, conversion to probable dementia, and mortality. Am J Alzheimers Dis Other Demen. 2021;36:15333175211012235. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Prince M, Bryce R, Ferri CP. World Alzheimer report 2011: the benefits of early diagnosis and intervention. Alzheimer’s Disease International. 2011. URL: https://www.alzint.org/u/WorldAlzheimerReport2011.pdf [accessed 2024-04-01]
• Patnode CD, Perdue LA, Rossom RC, Rushkin MC, Redmond N, Thomas RG, et al. Screening for cognitive impairment in older adults: updated evidence report and systematic review for the US preventive services task force. JAMA. Feb 25, 2020;323(8):764-785. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Canadian Task Force on Preventive Health Care, Pottie K, Rahal R, Jaramillo A, Birtwhistle R, Thombs BD, et al. Recommendations on screening for cognitive impairment in older adults. CMAJ. Jan 05, 2016;188(1):37-46. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tahami Monfared AA, Phan NT, Pearson I, Mauskopf J, Cho M, Zhang Q, et al. A systematic review of clinical practice guidelines for Alzheimer's disease and strategies for future advancements. Neurol Ther. Aug 2023;12(4):1257-1284. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mattke S, Jun H, Chen E, Liu Y, Becker A, Wallick C. Expected and diagnosed rates of mild cognitive impairment and dementia in the U.S. medicare population: observational analysis. Alzheimers Res Ther. Jul 22, 2023;15(1):128. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Manly JJ, Tang MX, Schupf N, Stern Y, Vonsattel JP, Mayeux R. Frequency and course of mild cognitive impairment in a multiethnic community. Ann Neurol. Apr 2008;63(4):494-506. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Black CM, Ambegaonkar BM, Pike J, Jones E, Husbands J, Khandker RK. The diagnostic pathway from cognitive impairment to dementia in Japan: quantification using real-world data. Alzheimer Dis Assoc Disord. 2019;33(4):346-353. [ CrossRef ] [ Medline ]
• Ritchie CW, Black CM, Khandker RK, Wood R, Jones E, Hu X, et al. Quantifying the diagnostic pathway for patients with cognitive impairment: real-world data from seven European and north American countries. J Alzheimers Dis. 2018;62(1):457-466. [ CrossRef ] [ Medline ]
• Folstein MF, Folstein SE, McHugh PR. "Mini-mental state". A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. Nov 1975;12(3):189-198. [ CrossRef ] [ Medline ]
• Tsoi KK, Chan JY, Hirai HW, Wong SY, Kwok TC. Cognitive tests to detect dementia: a systematic review and meta-analysis. JAMA Intern Med. Sep 2015;175(9):1450-1458. [ CrossRef ] [ Medline ]
• Lopez MN, Charter RA, Mostafavi B, Nibut LP, Smith WE. Psychometric properties of the Folstein mini-mental state examination. Assessment. Jun 2005;12(2):137-144. [ CrossRef ] [ Medline ]
• Nasreddine ZS, Phillips NA, Bédirian V, Charbonneau S, Whitehead V, Collin I, et al. The Montreal cognitive assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. Apr 2005;53(4):695-699. [ CrossRef ] [ Medline ]
• O'Caoimh R, Timmons S, Molloy DW. Screening for mild cognitive impairment: comparison of "MCI specific" screening instruments. J Alzheimers Dis. 2016;51(2):619-629. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Trzepacz PT, Hochstetler H, Wang S, Walker B, Saykin AJ, Alzheimer’s Disease Neuroimaging Initiative. Relationship between the Montreal cognitive assessment and mini-mental state examination for assessment of mild cognitive impairment in older adults. BMC Geriatr. Sep 07, 2015;15:107. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nasreddine ZS, Phillips N, Chertkow H. Normative data for the Montreal Cognitive Assessment (MoCA) in a population-based sample. Neurology. Mar 06, 2012;78(10):765-766. [ CrossRef ] [ Medline ]
• Monroe T, Carter M. Using the Folstein Mini Mental State Exam (MMSE) to explore methodological issues in cognitive aging research. Eur J Ageing. Sep 2012;9(3):265-274. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Damian AM, Jacobson SA, Hentz JG, Belden CM, Shill HA, Sabbagh MN, et al. The Montreal cognitive assessment and the mini-mental state examination as screening instruments for cognitive impairment: item analyses and threshold scores. Dement Geriatr Cogn Disord. 2011;31(2):126-131. [ CrossRef ] [ Medline ]
• Kaufer DI, Williams CS, Braaten AJ, Gill K, Zimmerman S, Sloane PD. Cognitive screening for dementia and mild cognitive impairment in assisted living: comparison of 3 tests. J Am Med Dir Assoc. Oct 2008;9(8):586-593. [ CrossRef ] [ Medline ]
• Gagnon C, Saillant K, Olmand M, Gayda M, Nigam A, Bouabdallaoui N, et al. Performances on the Montreal cognitive assessment along the cardiovascular disease continuum. Arch Clin Neuropsychol. Jan 17, 2022;37(1):117-124. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cooley SA, Heaps JM, Bolzenius JD, Salminen LE, Baker LM, Scott SE, et al. Longitudinal change in performance on the Montreal cognitive assessment in older adults. Clin Neuropsychol. 2015;29(6):824-835. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Caoimh R, Gao Y, McGlade C, Healy L, Gallagher P, Timmons S, et al. Comparison of the quick mild cognitive impairment (Qmci) screen and the SMMSE in screening for mild cognitive impairment. Age Ageing. Sep 2012;41(5):624-629. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Caoimh R, Molloy DW. Comparing the diagnostic accuracy of two cognitive screening instruments in different dementia subtypes and clinical depression. Diagnostics (Basel). Aug 08, 2019;9(3):93. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Clarnette R, O'Caoimh R, Antony DN, Svendrovski A, Molloy DW. Comparison of the Quick Mild Cognitive Impairment (Qmci) screen to the Montreal Cognitive Assessment (MoCA) in an Australian geriatrics clinic. Int J Geriatr Psychiatry. Jun 2017;32(6):643-649. [ CrossRef ] [ Medline ]
• Glynn K, Coen R, Lawlor BA. Is the Quick Mild Cognitive Impairment screen (QMCI) more accurate at detecting mild cognitive impairment than existing short cognitive screening tests? A systematic review of the current literature. Int J Geriatr Psychiatry. Dec 2019;34(12):1739-1746. [ CrossRef ] [ Medline ]
• Lee MT, Chang WY, Jang Y. Psychometric and diagnostic properties of the Taiwan version of the quick mild cognitive impairment screen. PLoS One. 2018;13(12):e0207851. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wallace SE, Donoso Brown EV, Simpson RC, D'Acunto K, Kranjec A, Rodgers M, et al. A comparison of electronic and paper versions of the Montreal cognitive assessment. Alzheimer Dis Assoc Disord. 2019;33(3):272-278. [ CrossRef ] [ Medline ]
• Gagnon C, Olmand M, Dupuy EG, Besnier F, Vincent T, Grégoire CA, et al. Videoconference version of the Montreal cognitive assessment: normative data for Quebec-French people aged 50 years and older. Aging Clin Exp Res. Jul 2022;34(7):1627-1633. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Friemel TN. The digital divide has grown old: determinants of a digital divide among seniors. New Media & Society. Jun 12, 2014;18(2):313-331. [ CrossRef ]
• Ventola CL. Mobile devices and apps for health care professionals: uses and benefits. P T. May 2014;39(5):356-364. [ FREE Full text ] [ Medline ]
• Searles C, Farnsworth JL, Jubenville C, Kang M, Ragan B. Test–retest reliability of the BrainFx 360® performance assessment. Athl Train Sports Health Care. Jul 2019;11(4):183-191. [ CrossRef ]
• Jones C, Miguel-Cruz A, Brémault-Phillips S. Technology acceptance and usability of the BrainFx SCREEN in Canadian military members and veterans with posttraumatic stress disorder and mild traumatic brain injury: mixed methods UTAUT study. JMIR Rehabil Assist Technol. May 13, 2021;8(2):e26078. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McMurray J, Levy A, Holyoke P. Psychometric evaluation and workflow integration study of a tablet-based tool to detect mild cognitive impairment in older adults: protocol for a mixed methods study. JMIR Res Protoc. May 21, 2021;10(5):e25520. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wilansky P, Eklund JM, Milner T, Kreindler D, Cheung A, Kovacs T, et al. Cognitive behavior therapy for anxious and depressed youth: improving homework adherence through mobile technology. JMIR Res Protoc. Nov 10, 2016;5(4):e209. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ammenwerth E, Iller C, Mahler C. IT-adoption and the interaction of task, technology and individuals: a fit framework and a case study. BMC Med Inform Decis Mak. Jan 09, 2006;6:3. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Goodhue DL, Thompson RL. Task-technology fit and individual performance. MIS Q. Jun 1995;19(2):213-236. [ CrossRef ]
• Beuscher L, Grando VT. Challenges in conducting qualitative research with individuals with dementia. Res Gerontol Nurs. Jan 2009;2(1):6-11. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Howe E. Informed consent, participation in research, and the Alzheimer's patient. Innov Clin Neurosci. May 2012;9(5-6):47-51. [ FREE Full text ] [ Medline ]
• Thorogood A, Mäki-Petäjä-Leinonen A, Brodaty H, Dalpé G, Gastmans C, Gauthier S, et al. Global Alliance for GenomicsHealth‚ AgeingDementia Task Team. Consent recommendations for research and international data sharing involving persons with dementia. Alzheimers Dement. Oct 2018;14(10):1334-1343. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Demeyere N, Haupt M, Webb SS, Strobel L, Milosevich ET, Moore MJ, et al. Introducing the tablet-based Oxford Cognitive Screen-Plus (OCS-Plus) as an assessment tool for subtle cognitive impairments. Sci Rep. Apr 12, 2021;11(1):8000. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nasreddine ZS, Patel BB. Validation of Montreal cognitive assessment, MoCA, alternate French versions. Can J Neurol Sci. Sep 2016;43(5):665-671. [ CrossRef ] [ Medline ]
• Mueller AE, Segal DL, Gavett B, Marty MA, Yochim B, June A, et al. Geriatric anxiety scale: item response theory analysis, differential item functioning, and creation of a ten-item short form (GAS-10). Int Psychogeriatr. Jul 2015;27(7):1099-1111. [ CrossRef ] [ Medline ]
• Segal DL, June A, Payne M, Coolidge FL, Yochim B. Development and initial validation of a self-report assessment tool for anxiety among older adults: the Geriatric Anxiety Scale. J Anxiety Disord. Oct 2010;24(7):709-714. [ CrossRef ] [ Medline ]
• Balsamo M, Cataldi F, Carlucci L, Fairfield B. Assessment of anxiety in older adults: a review of self-report measures. Clin Interv Aging. 2018;13:573-593. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gatti A, Gottschling J, Brugnera A, Adorni R, Zarbo C, Compare A, et al. An investigation of the psychometric properties of the Geriatric Anxiety Scale (GAS) in an Italian sample of community-dwelling older adults. Aging Ment Health. Sep 2018;22(9):1170-1178. [ CrossRef ] [ Medline ]
• Yochim BP, Mueller AE, June A, Segal DL. Psychometric properties of the Geriatric Anxiety Scale: comparison to the beck anxiety inventory and geriatric anxiety inventory. Clin Gerontol. Dec 06, 2010;34(1):21-33. [ CrossRef ]
• Recent concussion (< 6 months ago) analysis result. Daisy Intelligence. 2016. URL: https://www.daisyintelligence.com/retail-solutions/ [accessed 2024-04-01]
• Malloy DW, O'Caoimh R. The Quick Guide: Scoring and Administration Instructions for The Quick Mild Cognitive Impairment (Qmci) Screen. Waterford, Ireland. Newgrange Press; 2017.
• O'Caoimh R, Gao Y, Svendovski A, Gallagher P, Eustace J, Molloy DW. Comparing approaches to optimize cut-off scores for short cognitive screening instruments in mild cognitive impairment and dementia. J Alzheimers Dis. 2017;57(1):123-133. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Breton A, Casey D, Arnaoutoglou NA. Cognitive tests for the detection of mild cognitive impairment (MCI), the prodromal stage of dementia: meta-analysis of diagnostic accuracy studies. Int J Geriatr Psychiatry. Feb 2019;34(2):233-242. [ CrossRef ] [ Medline ]
• Umemneku Chikere CM, Wilson K, Graziadio S, Vale L, Allen AJ. Diagnostic test evaluation methodology: a systematic review of methods employed to evaluate diagnostic tests in the absence of gold standard - An update. PLoS One. 2019;14(10):e0223832. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Espinosa A, Alegret M, Boada M, Vinyes G, Valero S, Martínez-Lage P, et al. Ecological assessment of executive functions in mild cognitive impairment and mild Alzheimer's disease. J Int Neuropsychol Soc. Sep 2009;15(5):751-757. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hawkins DM, Garrett JA, Stephenson B. Some issues in resolution of diagnostic tests using an imperfect gold standard. Stat Med. Jul 15, 2001;20(13):1987-2001. [ CrossRef ] [ Medline ]
• Hadgu A, Dendukuri N, Hilden J. Evaluation of nucleic acid amplification tests in the absence of a perfect gold-standard test: a review of the statistical and epidemiologic issues. Epidemiology. Sep 2005;16(5):604-612. [ CrossRef ] [ Medline ]
• Marx RG, Menezes A, Horovitz L, Jones EC, Warren RF. A comparison of two time intervals for test-retest reliability of health status instruments. J Clin Epidemiol. Aug 2003;56(8):730-735. [ CrossRef ] [ Medline ]
• Paiva CE, Barroso EM, Carneseca EC, de Pádua Souza C, Dos Santos FT, Mendoza López RV, et al. A critical analysis of test-retest reliability in instrument validation studies of cancer patients under palliative care: a systematic review. BMC Med Res Methodol. Jan 21, 2014;14:8. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Streiner DL, Kottner J. Recommendations for reporting the results of studies of instrument and scale development and testing. J Adv Nurs. Sep 2014;70(9):1970-1979. [ CrossRef ] [ Medline ]
• Streiner DL. A checklist for evaluating the usefulness of rating scales. Can J Psychiatry. Mar 1993;38(2):140-148. [ CrossRef ] [ Medline ]
• Peyre H, Leplège A, Coste J. Missing data methods for dealing with missing items in quality of life questionnaires. A comparison by simulation of personal mean score, full information maximum likelihood, multiple imputation, and hot deck techniques applied to the SF-36 in the French 2003 decennial health survey. Qual Life Res. Mar 2011;20(2):287-300. [ CrossRef ] [ Medline ]
• Nevado-Holgado AJ, Kim CH, Winchester L, Gallacher J, Lovestone S. Commonly prescribed drugs associate with cognitive function: a cross-sectional study in UK Biobank. BMJ Open. Nov 30, 2016;6(11):e012177. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moore AR, O'Keeffe ST. Drug-induced cognitive impairment in the elderly. Drugs Aging. Jul 1999;15(1):15-28. [ CrossRef ] [ Medline ]
• Rogers J, Wiese BS, Rabheru K. The older brain on drugs: substances that may cause cognitive impairment. Geriatr Aging. 2008;11(5):284-289. [ FREE Full text ]
• Marvanova M. Drug-induced cognitive impairment: effect of cardiovascular agents. Ment Health Clin. Jul 2016;6(4):201-206. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Espeland MA, Rapp SR, Manson JE, Goveas JS, Shumaker SA, Hayden KM, et al. WHIMSYWHIMS-ECHO Study Groups. Long-term effects on cognitive trajectories of postmenopausal hormone therapy in two age groups. J Gerontol A Biol Sci Med Sci. Jun 01, 2017;72(6):838-845. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Luis CA, Keegan AP, Mullan M. Cross validation of the Montreal cognitive assessment in community dwelling older adults residing in the Southeastern US. Int J Geriatr Psychiatry. Feb 2009;24(2):197-201. [ CrossRef ] [ Medline ]
• Cunje A, Molloy DW, Standish TI, Lewis DL. Alternate forms of logical memory and verbal fluency tasks for repeated testing in early cognitive changes. Int Psychogeriatr. Feb 2007;19(1):65-75. [ CrossRef ] [ Medline ]
• Molloy DW, Standish TI, Lewis DL. Screening for mild cognitive impairment: comparing the SMMSE and the ABCS. Can J Psychiatry. Jan 2005;50(1):52-58. [ CrossRef ] [ Medline ]
• Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. 4th edition. Oxford, UK. Oxford University Press; 2008.
• Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. Jun 2016;15(2):155-163. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-282. [ FREE Full text ] [ Medline ]
• Zhuang L, Yang Y, Gao J. Cognitive assessment tools for mild cognitive impairment screening. J Neurol. May 2021;268(5):1615-1622. [ CrossRef ] [ Medline ]
• Zhang J, Wang L, Deng X, Fei G, Jin L, Pan X, et al. Five-minute cognitive test as a new quick screening of cognitive impairment in the elderly. Aging Dis. Dec 2019;10(6):1258-1269. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Feldman HH, Jacova C, Robillard A, Garcia A, Chow T, Borrie M, et al. Diagnosis and treatment of dementia: 2. Diagnosis. CMAJ. Mar 25, 2008;178(7):825-836. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sabbagh MN, Boada M, Borson S, Chilukuri M, Dubois B, Ingram J, et al. Early detection of mild cognitive impairment (MCI) in primary care. J Prev Alzheimers Dis. 2020;7(3):165-170. [ CrossRef ] [ Medline ]
• Milne A. Dementia screening and early diagnosis: the case for and against. Health Risk Soc. Mar 05, 2010;12(1):65-76. [ CrossRef ]
• Screening tools to identify adults with cognitive impairment associated with dementia: diagnostic accuracy. Canadian Agency for Drugs and Technologies in Health. 2014. URL: https:/​/www.​cadth.ca/​sites/​default/​files/​pdf/​htis/​nov-2014/​RB0752%20Cognitive%20Assessments%20for%20Dementia%20Final.​pdf [accessed 2024-04-01]
• Chehrehnegar N, Nejati V, Shati M, Rashedi V, Lotfi M, Adelirad F, et al. Early detection of cognitive disturbances in mild cognitive impairment: a systematic review of observational studies. Psychogeriatrics. Mar 2020;20(2):212-228. [ CrossRef ] [ Medline ]
• Chan JY, Yau ST, Kwok TC, Tsoi KK. Diagnostic performance of digital cognitive tests for the identification of MCI and dementia: a systematic review. Ageing Res Rev. Dec 2021;72:101506. [ CrossRef ] [ Medline ]
• Cubillos C, Rienzo A. Digital cognitive assessment tests for older adults: systematic literature review. JMIR Ment Health. Dec 08, 2023;10:e47487. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chen R, Foschini L, Kourtis L, Signorini A, Jankovic F, Pugh M, et al. Developing measures of cognitive impairment in the real world from consumer-grade multimodal sensor streams. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019. Presented at: KDD '19; August 4-8, 2019;2145; Anchorage, AK. URL: https://dl.acm.org/doi/10.1145/3292500.3330690 [ CrossRef ]
• Koo BM, Vizer LM. Mobile technology for cognitive assessment of older adults: a scoping review. Innov Aging. Jan 2019;3(1):igy038. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zygouris S, Ntovas K, Giakoumis D, Votis K, Doumpoulakis S, Segkouli S, et al. A preliminary study on the feasibility of using a virtual reality cognitive training application for remote detection of mild cognitive impairment. J Alzheimers Dis. 2017;56(2):619-627. [ CrossRef ] [ Medline ]
• Liu Q, Song H, Yan M, Ding Y, Wang Y, Chen L, et al. Virtual reality technology in the detection of mild cognitive impairment: a systematic review and meta-analysis. Ageing Res Rev. Jun 2023;87:101889. [ CrossRef ] [ Medline ]
• Fayemiwo MA, Olowookere TA, Olaniyan OO, Ojewumi TO, Oyetade IS, Freeman S, et al. Immediate word recall in cognitive assessment can predict dementia using machine learning techniques. Alzheimers Res Ther. Jun 15, 2023;15(1):111. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Statsenko Y, Meribout S, Habuza T, Almansoori TM, van Gorkom KN, Gelovani JG, et al. Patterns of structure-function association in normal aging and in Alzheimer's disease: screening for mild cognitive impairment and dementia with ML regression and classification models. Front Aging Neurosci. 2022;14:943566. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Roebuck-Spencer TM, Glen T, Puente AE, Denney RL, Ruff RM, Hostetter G, et al. Cognitive screening tests versus comprehensive neuropsychological test batteries: a national academy of neuropsychology education paper†. Arch Clin Neuropsychol. Jun 01, 2017;32(4):491-498. [ CrossRef ] [ Medline ]
• Jammeh EA, Carroll CB, Pearson SW, Escudero J, Anastasiou A, Zhao P, et al. Machine-learning based identification of undiagnosed dementia in primary care: a feasibility study. BJGP Open. Jul 2018;2(2):bjgpopen18X101589. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Riello M, Rusconi E, Treccani B. The role of brief global cognitive tests and neuropsychological expertise in the detection and differential diagnosis of dementia. Front Aging Neurosci. 2021;13:648310. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McDonnell M, Dill L, Panos S, Amano S, Brown W, Giurgius S, et al. Verbal fluency as a screening tool for mild cognitive impairment. Int Psychogeriatr. Sep 2020;32(9):1055-1062. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wojtowicz A, Larner AJ. Diagnostic test accuracy of cognitive screeners in older people. Prog Neurol Psychiatry. Mar 20, 2017;21(1):17-21. [ CrossRef ]
• Larner AJ. Cognitive screening instruments for the diagnosis of mild cognitive impairment. Prog Neurol Psychiatry. Apr 07, 2016;20(2):21-26. [ CrossRef ]
• Heintz BD, Keenan KG. Spiral tracing on a touchscreen is influenced by age, hand, implement, and friction. PLoS One. 2018;13(2):e0191309. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Laguna K, Babcock RL. Computer anxiety in young and older adults: implications for human-computer interactions in older populations. Comput Human Behav. Aug 1997;13(3):317-326. [ CrossRef ]
• Wild KV, Mattek NC, Maxwell SA, Dodge HH, Jimison HB, Kaye JA. Computer-related self-efficacy and anxiety in older adults with and without mild cognitive impairment. Alzheimers Dement. Nov 2012;8(6):544-552. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wiechmann D, Ryan AM. Reactions to computerized testing in selection contexts. Int J Sel Assess. Jul 30, 2003;11(2-3):215-229. [ CrossRef ]
• Gass CS, Curiel RE. Test anxiety in relation to measures of cognitive and intellectual functioning. Arch Clin Neuropsychol. Aug 2011;26(5):396-404. [ CrossRef ] [ Medline ]
• Barbic D, Kim B, Salehmohamed Q, Kemplin K, Carpenter CR, Barbic SP. Diagnostic accuracy of the Ottawa 3DY and short blessed test to detect cognitive dysfunction in geriatric patients presenting to the emergency department. BMJ Open. Mar 16, 2018;8(3):e019652. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Owens AP, Ballard C, Beigi M, Kalafatis C, Brooker H, Lavelle G, et al. Implementing remote memory clinics to enhance clinical care during and after COVID-19. Front Psychiatry. 2020;11:579934. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Geddes MR, O'Connell ME, Fisk JD, Gauthier S, Camicioli R, Ismail Z, et al. Alzheimer Society of Canada Task Force on Dementia Care Best Practices for COVID‐19. Remote cognitive and behavioral assessment: report of the Alzheimer Society of Canada task force on dementia care best practices for COVID-19. Alzheimers Dement (Amst). 2020;12(1):e12111. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by G Eysenbach, T de Azevedo Cardoso; submitted 29.01.24; peer-reviewed by J Gao, MJ Moore; comments to author 20.02.24; revised version received 05.03.24; accepted 19.03.24; published 19.04.24.

©Josephine McMurray, AnneMarie Levy, Wei Pang, Paul Holyoke. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 19.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

More From Forbes

5 tips to enhance your research paper’s visibility and altmetric score.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

US evangelist Billy Graham addresses a crowd of football supporters at Stamford Bridge, London, ... [+] during half-time at the match between Chelsea and Newcastle United. (Photo by Edward Miller/Getty Images)

I previously wrote about the importance of attracting public attention to scientific research . In today’s world, where billions of people are attached to their digital devices watching the very addictive but often useless TikTok content or receiving instant gratification by engaging in meaningless debates about celebrities, scientists need to find creative ways to have their research noticed. Popularizing scientific research helps inspire the younger generations to go into science and provide the general public with a sense of optimism enabling the government to channel more resources into science. People do need inspiration. But very often, even very important scientific breakthroughs requiring many years, hard work, skill, funding, and genuine serendipity go largely unnoticed by the general public.

One of the best ways to measure expert and public attention is the cumulative Altmetric Attention Score , originally developed by Digital Science and adopted by many prestigious publishers, including Nature Publishing Group. Every Nature paper and the papers published by pretty much every credible publisher are tracked by Digital Science by the Document Object Identification (DOI) or the Unique Resource Locator (URL) . While Altmetric has many limitations, for example, it does not track LinkedIn posts and may not adequately cover the impact of top-tier media coverage, at the moment it is the blueprint for tracking attention.

Altmetric Score in The Age of Generative AI

Media attention is likely to be very important in the age of generative AI. Many modern generative systems, such as ChatGPT, Claude, Mistral, and Gemini, as well as hundreds of Large Language Models (LLMs) in China, use the data from the same sources referenced in Altmetric to learn. The more times generative systems see the same concept presented in different contexts, the better they learn. So if you want to contribute to the training of AI systems that may thank you for it in the future - Altmetric is the way to go.

So what can a research group do to ensure they are communicating their findings effectively and increasing the visibility of their research to ensure it gets reflected in the Altmetric Attention Score?

JPMorgan Joins Goldman Sachs In Serious Bitcoin Halving Price Warning

Google makes a major new sale offer to pixel 8 buyers, the crazy true story behind netflix s baby reindeer what happens to martha.

Altmetric openly discloses the weights of the various sources and the scoring algorithm is relatively straightforward. It is easy to learn, and there are multiple online resources providing advice on how to share your research in ways that will be captured by Altmetric. Cambridge University Press published a guideline to Altmetric for the authors on how to popularize their research with Altmetric in mind. Wolters Kluwer put out a guide and the editor of Toxicology and Pathology wrote a comprehensive overview of Altmetric and how to use it. Surprisingly, this overview got an Altmetric Attention Score of only 4 at the time of the writing, but was cited 137 times according to Google Scholar .

Altmetric monitors social networks, including X (formerly Twitter), Facebook, and Reddit; all major top-tier mainstream media, mainstream science blogs, policy documents, patents, Wikipedia articles, peer review websites, F1000, Syllabi, X (formerly Twitter), tracked Facebook pages, Reddit, one of the Stack Exchange sites, and Youtube. Unfortunately, several powerful platforms, including LinkedIn, are not currently tracked.

The popularity of the paper depends on many factors. Firstly, it has to be novel, trendy, and newsworthy. You are unlikely to get high Altmetric Score with a boring topic. Secondly, papers coming out of popular labs in top-tier academic institutions and in top journals are likely to attract more attention. Often, the communications officers in these academic institutions work closely with the media to amplify notable research. Celebrity companies, for example, Google DeepMind, consistently get higher coverage.

Screenshot of the Altmetric Attention Score "Flower" showing several tracked sources

Here are the five tips for increasing the visibility of your work and ensuring that reach is tracked and reflected by Altmetric:

1. Understand How Altmetric System Works

Congratulations, if you read this article and looked at what sources are tracked by Altmetric. Most likely, you got the basics and will be able to get a “balanced flower” by making a press release, tweeting the DOI of the paper on X, posting a video overview of your paper on Youtube, announcing on Reddit (I still need to learn how to do this).

To understand how Altmetric works, I emailed a few questions to Miguel Garcia, Director of Product and Data Analytics Hub at Digital Science and my first question was wether the Altmetric algorithm is open source. “The Altmetric Attention Score's calculation is not open source but we try to provide as much information as possible around how we calculate it here, and are currently considering what steps we might take to make our algorithms more transparent.” He also provided a link to how the Altmetric Attention Score is calculated.

Many professionals use LinkedIn as the primary social media resource and I was wondering why Altmetric stopped tracking it. Bad news - technical reasons prevent tracking DOIs on LinkedIn. Good news - they are actively seeking ways to appropriately track mentions on LinkedIn and we may see some news toward the end of the year.

My other big question was how does Altmetric count tweets and retweets on X. What if there are many posts from the same account? Miguel’s response was: “Re-tweets count less than original tweets. In addition to that, modifiers are applied to the type of account that is tweeting in order to reduce the weight of the tweet in situations where we find signals of bias or promiscuity (for example a journal publisher only tweeting their own articles). Besides that, we have conditions around the maximum number of retweets in order to limit the maximum impact they would have.”

So tweeting the article many times will not help you. But if other scientists tweet you paper with a DOI - these tweets will get counted. So tweet others as you would like to be tweeted.

2. Make a Press Release and Distribute to Science-focused Media

If your paper is significant, for example, you elucidated novel disease biology, discovered a new drug, developed a new fancy algorithm, designed a new material, or developed a new application for a quantum computer, it is worthwhile investing some time and resources in writing a press release. If you are working for an academic institution, most likely they have a communications office that will help you. If you do not have this luxury, you will need to learn how to write a press release. Plenty of free online guides cover the basics of press release writing. And press releases are one area where ChatGPT and other generative tools do surprisingly well. Upload your paper and ask it to write a press release, check for errors or exaggerations, edit, and you are ready to go. Just make sure to include the DOI and the URL of your paper. A proper business press release on BusinessWire or PRNewswire may cost several thousand dollars. In my opinion, these resources are dramatically overcharging while providing little service. I don't remember a case where a journalist picked up our news based on a commercial press release. But these releases are often reposted by other online press release distributors and the boost to Altmetric may be considerable. The default news release distribution service for research news is EurekAlert. This resource may sometimes result in journalistic coverage as many reporters are using it for science news. There are many free resources you can use if you do not have any budget.

Once the press release is issued, share it with the media. Share the resulting news coverage via your social networks and contacts. Many journalists track the popularity of their news articles and giving them several thousand extra views from professional audience and increasing their social following increases the chances that they will cover the next important research paper.

3. Make a Blog Post

Writing a blog post can be longer and more comprehensive than the press release. Make sure to add fancy diagrams and graphical explainers. You can share the blog post with the journalists at the same time as the press release. Your blog may serve as a source of inspiration for third party news coverage. Make sure to reference the DOI and URL of your paper.

If your paper is in one of the Nature journals, consider writing a “Behind the Paper" blog post on Nature Bioengineering Community. Surprisingly, these blogs are rarely picked up by Altmetric but may serve as a source of inspiration for the journalists and social media influencers. Plus, it is a resource by the Nature Publishing Group.

4. Tweet and Ask Your Team Members to Tweet

Each post on X gives you a quarter of an Altmetric point. If your paper goes viral on X, your Altmetric score can be considerable. Plus, once journalists notice that it went viral, they will be more likely to cover the story, further increasing the score.

Try to choose the time of the post, the hashtags, and the images wisely. Since Elon Musk took over X and opened the algorithm it became very transparent and easy to optimize for. Here are the top 10 tips for boosting attention for a post on X. Make sure to include the DOI or the URL of the paper for Altmetric to find the post.

5. Experiment, Learn, Repeat

My highest Altmetric Attention Score core to date was around 1,500 for a paper in Nature Biotechnology published in 2019, where we used a novel method for designing small molecules called Generative Tensorial Reinforcement Learning (GENTRL) to generate new molecules with druglike properties that got synthesized and tested all the way into mice. In 2024, we went further and showed that an AI-generated molecule for an AI-discovered target was tested all the way up to Phase II human trials, but the paper published in Nature Biotechnology, let’s call it the TNIK paper , has achieved a score ofjust over 600 to date. So what has changed and what can we learn from these two papers?

The popularity of the paper depends on many factors. Ones which capture the public imagination or have widespread appeal are of course, much more likely to gain traction online. When we published the GENTRL paper in 2019, Generative AI was in its infancy, and there are pretty much no other companies that I heard of at the intersection of generative AI and drug discovery. We also published multiple articles in this field in the years leading to that paper and many key opinion leaders (KOLs) followed us. That following included a small army of generative AI skeptics who not only contributed to multiple rejections in peer-reviewed journals but also openly criticized this approach in social networks. This criticism also helped boost the Altmetric Score and bring more attention to the study. So first learning from this exercise - negative publicity helps overall publicity. As long as you are certain that your research results are honest - leave room for criticism and even help expose your paper’s weaknesses. Critics are your greatest Altmetric boosters. Since readers and, by extension journalists, react to negative news and drama stronger than to positive news, critical reviews will boost your Altmetric as long as the DOI or URL of the paper is properly referenced.

Secondly, papers coming out of popular labs in top-tier academic institutions and in top journals are likely to attract more attention. Often, the communications officers in these academic institutions work closely with the media to amplify notable research. Celebrity companies, for example, Google DeepMind, always get a higher level of coverage. For example, the AlphaFold paper published in July 2021 in Nature got an Altmetric Attention Score of over 3,500 . Even though I have not seen any drugs out of AlphaFold reaching preclinical candidate status, I predict the popularity of this tool will result in the first Nobel Prize in this area. Therefore, in order to become famous and popularize your research more effectively, it is a good idea to build up the public profile of yourself and your work. For example, Kardashians are famous for being famous .

Be careful with Wikipedia. I made a mistake explaining the importance of Wikipedia to students when lecturing on the future of generative AI, and one or two of them got banned for expanding the articles with paper references. Wikipedia requires that these are added by independent editors rather than the authors of papers themselves, but if some editors do not like it, they will not go deep or investigate - they will assume wrongdoing. So it is better to avoid even talking about Wikipedia. References there should happen naturally and often some of the more popular papers get picked up and referenced by veteran editors.

Experimenting with Altmetric will also help you explore new strategies for popularizing scientific research and develop new strategies for inspiring people to learn or even get into the new exciting field. UNESCO estimates that there was just over 8 million full-time equivalent (FTE) researchers in 2018 globally. Only a fraction of these are in biotechnology - less than 0.01% of the global population. If you motivate a million students to go into biotechnology by popularizing your research, you double this number.

• Editorial Standards
• Reprints & Permissions