augmented reality thesis subject

MIT Libraries home DSpace@MIT

• DSpace@MIT Home
• MIT Libraries
• Doctoral Theses

Enhancing internet of things experience in augmented reality environments

Other Contributors

Terms of use, description, date issued, collections.

• CSE PhD Theses

ORIGINAL RESEARCH article

The past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literature.

• 1 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, Milan, Italy
• 2 Department of Psychology, Catholic University of the Sacred Heart, Milan, Italy
• 3 Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, Valencia, Spain

The recent appearance of low cost virtual reality (VR) technologies – like the Oculus Rift, the HTC Vive and the Sony PlayStation VR – and Mixed Reality Interfaces (MRITF) – like the Hololens – is attracting the attention of users and researchers suggesting it may be the next largest stepping stone in technological innovation. However, the history of VR technology is longer than it may seem: the concept of VR was formulated in the 1960s and the first commercial VR tools appeared in the late 1980s. For this reason, during the last 20 years, 100s of researchers explored the processes, effects, and applications of this technology producing 1000s of scientific papers. What is the outcome of this significant research work? This paper wants to provide an answer to this question by exploring, using advanced scientometric techniques, the existing research corpus in the field. We collected all the existent articles about VR in the Web of Science Core Collection scientific database, and the resultant dataset contained 21,667 records for VR and 9,944 for augmented reality (AR). The bibliographic record contained various fields, such as author, title, abstract, country, and all the references (needed for the citation analysis). The network and cluster analysis of the literature showed a composite panorama characterized by changes and evolutions over the time. Indeed, whether until 5 years ago, the main publication media on VR concerned both conference proceeding and journals, more recently journals constitute the main medium of communication. Similarly, if at first computer science was the leading research field, nowadays clinical areas have increased, as well as the number of countries involved in VR research. The present work discusses the evolution and changes over the time of the use of VR in the main areas of application with an emphasis on the future expected VR’s capacities, increases and challenges. We conclude considering the disruptive contribution that VR/AR/MRITF will be able to get in scientific fields, as well in human communication and interaction, as already happened with the advent of mobile phones by increasing the use and the development of scientific applications (e.g., in clinical areas) and by modifying the social communication and interaction among people.

Introduction

In the last 5 years, virtual reality (VR) and augmented reality (AR) have attracted the interest of investors and the general public, especially after Mark Zuckerberg bought Oculus for two billion dollars ( Luckerson, 2014 ; Castelvecchi, 2016 ). Currently, many other companies, such as Sony, Samsung, HTC, and Google are making huge investments in VR and AR ( Korolov, 2014 ; Ebert, 2015 ; Castelvecchi, 2016 ). However, if VR has been used in research for more than 25 years, and now there are 1000s of papers and many researchers in the field, comprising a strong, interdisciplinary community, AR has a more recent application history ( Burdea and Coiffet, 2003 ; Kim, 2005 ; Bohil et al., 2011 ; Cipresso and Serino, 2014 ; Wexelblat, 2014 ). The study of VR was initiated in the computer graphics field and has been extended to several disciplines ( Sutherland, 1965 , 1968 ; Mazuryk and Gervautz, 1996 ; Choi et al., 2015 ). Currently, videogames supported by VR tools are more popular than the past, and they represent valuables, work-related tools for neuroscientists, psychologists, biologists, and other researchers as well. Indeed, for example, one of the main research purposes lies from navigation studies that include complex experiments that could be done in a laboratory by using VR, whereas, without VR, the researchers would have to go directly into the field, possibly with limited use of intervention. The importance of navigation studies for the functional understanding of human memory in dementia has been a topic of significant interest for a long time, and, in 2014, the Nobel Prize in “Physiology or Medicine” was awarded to John M. O’Keefe, May-Britt Moser, and Edvard I. Moser for their discoveries of nerve cells in the brain that enable a sense of place and navigation. Journals and magazines have extended this knowledge by writing about “the brain GPS,” which gives a clear idea of the mechanism. A huge number of studies have been conducted in clinical settings by using VR ( Bohil et al., 2011 ; Serino et al., 2014 ), and Nobel Prize winner, Edvard I. Moser commented about the use of VR ( Minderer et al., 2016 ), highlighting its importance for research and clinical practice. Moreover, the availability of free tools for VR experimental and computational use has made it easy to access any field ( Riva et al., 2011 ; Cipresso, 2015 ; Brown and Green, 2016 ; Cipresso et al., 2016 ).

Augmented reality is a more recent technology than VR and shows an interdisciplinary application framework, in which, nowadays, education and learning seem to be the most field of research. Indeed, AR allows supporting learning, for example increasing-on content understanding and memory preservation, as well as on learning motivation. However, if VR benefits from clear and more definite fields of application and research areas, AR is still emerging in the scientific scenarios.

In this article, we present a systematic and computational analysis of the emerging interdisciplinary VR and AR fields in terms of various co-citation networks in order to explore the evolution of the intellectual structure of this knowledge domain over time.

Virtual Reality Concepts and Features

The concept of VR could be traced at the mid of 1960 when Ivan Sutherland in a pivotal manuscript attempted to describe VR as a window through which a user perceives the virtual world as if looked, felt, sounded real and in which the user could act realistically ( Sutherland, 1965 ).

Since that time and in accordance with the application area, several definitions have been formulated: for example, Fuchs and Bishop (1992) defined VR as “real-time interactive graphics with 3D models, combined with a display technology that gives the user the immersion in the model world and direct manipulation” ( Fuchs and Bishop, 1992 ); Gigante (1993) described VR as “The illusion of participation in a synthetic environment rather than external observation of such an environment. VR relies on a 3D, stereoscopic head-tracker displays, hand/body tracking and binaural sound. VR is an immersive, multi-sensory experience” ( Gigante, 1993 ); and “Virtual reality refers to immersive, interactive, multi-sensory, viewer-centered, 3D computer generated environments and the combination of technologies required building environments” ( Cruz-Neira, 1993 ).

As we can notice, these definitions, although different, highlight three common features of VR systems: immersion, perception to be present in an environment, and interaction with that environment ( Biocca, 1997 ; Lombard and Ditton, 1997 ; Loomis et al., 1999 ; Heeter, 2000 ; Biocca et al., 2001 ; Bailenson et al., 2006 ; Skalski and Tamborini, 2007 ; Andersen and Thorpe, 2009 ; Slater, 2009 ; Sundar et al., 2010 ). Specifically, immersion concerns the amount of senses stimulated, interactions, and the reality’s similarity of the stimuli used to simulate environments. This feature can depend on the properties of the technological system used to isolate user from reality ( Slater, 2009 ).

Higher or lower degrees of immersion can depend by three types of VR systems provided to the user:

• Non-immersive systems are the simplest and cheapest type of VR applications that use desktops to reproduce images of the world.

• Immersive systems provide a complete simulated experience due to the support of several sensory outputs devices such as head mounted displays (HMDs) for enhancing the stereoscopic view of the environment through the movement of the user’s head, as well as audio and haptic devices.

• Semi-immersive systems such as Fish Tank VR are between the two above. They provide a stereo image of a three dimensional (3D) scene viewed on a monitor using a perspective projection coupled to the head position of the observer ( Ware et al., 1993 ). Higher technological immersive systems have showed a closest experience to reality, giving to the user the illusion of technological non-mediation and feeling him or her of “being in” or present in the virtual environment ( Lombard and Ditton, 1997 ). Furthermore, higher immersive systems, than the other two systems, can give the possibility to add several sensory outputs allowing that the interaction and actions were perceived as real ( Loomis et al., 1999 ; Heeter, 2000 ; Biocca et al., 2001 ).

Finally, the user’s VR experience could be disclosed by measuring presence, realism, and reality’s levels. Presence is a complex psychological feeling of “being there” in VR that involves the sensation and perception of physical presence, as well as the possibility to interact and react as if the user was in the real world ( Heeter, 1992 ). Similarly, the realism’s level corresponds to the degree of expectation that the user has about of the stimuli and experience ( Baños et al., 2000 , 2009 ). If the presented stimuli are similar to reality, VR user’s expectation will be congruent with reality expectation, enhancing VR experience. In the same way, higher is the degree of reality in interaction with the virtual stimuli, higher would be the level of realism of the user’s behaviors ( Baños et al., 2000 , 2009 ).

From Virtual to Augmented Reality

Looking chronologically on VR and AR developments, we can trace the first 3D immersive simulator in 1962, when Morton Heilig created Sensorama, a simulated experience of a motorcycle running through Brooklyn characterized by several sensory impressions, such as audio, olfactory, and haptic stimuli, including also wind to provide a realist experience ( Heilig, 1962 ). In the same years, Ivan Sutherland developed The Ultimate Display that, more than sound, smell, and haptic feedback, included interactive graphics that Sensorama didn’t provide. Furthermore, Philco developed the first HMD that together with The Sword of Damocles of Sutherland was able to update the virtual images by tracking user’s head position and orientation ( Sutherland, 1965 ). In the 70s, the University of North Carolina realized GROPE, the first system of force-feedback and Myron Krueger created VIDEOPLACE an Artificial Reality in which the users’ body figures were captured by cameras and projected on a screen ( Krueger et al., 1985 ). In this way two or more users could interact in the 2D-virtual space. In 1982, the US’ Air Force created the first flight simulator [Visually Coupled Airbone System Simulator (VCASS)] in which the pilot through an HMD could control the pathway and the targets. Generally, the 80’s were the years in which the first commercial devices began to emerge: for example, in 1985 the VPL company commercialized the DataGlove, glove sensors’ equipped able to measure the flexion of fingers, orientation and position, and identify hand gestures. Another example is the Eyephone, created in 1988 by the VPL Company, an HMD system for completely immerging the user in a virtual world. At the end of 80’s, Fake Space Labs created a Binocular-Omni-Orientational Monitor (BOOM), a complex system composed by a stereoscopic-displaying device, providing a moving and broad virtual environment, and a mechanical arm tracking. Furthermore, BOOM offered a more stable image and giving more quickly responses to movements than the HMD devices. Thanks to BOOM and DataGlove, the NASA Ames Research Center developed the Virtual Wind Tunnel in order to research and manipulate airflow in a virtual airplane or space ship. In 1992, the Electronic Visualization Laboratory of the University of Illinois created the CAVE Automatic Virtual Environment, an immersive VR system composed by projectors directed on three or more walls of a room.

More recently, many videogames companies have improved the development and quality of VR devices, like Oculus Rift, or HTC Vive that provide a wider field of view and lower latency. In addition, the actual HMD’s devices can be now combined with other tracker system as eye-tracking systems (FOVE), and motion and orientation sensors (e.g., Razer Hydra, Oculus Touch, or HTC Vive).

Simultaneously, at the beginning of 90’, the Boing Corporation created the first prototype of AR system for showing to employees how set up a wiring tool ( Carmigniani et al., 2011 ). At the same time, Rosenberg and Feiner developed an AR fixture for maintenance assistance, showing that the operator performance enhanced by added virtual information on the fixture to repair ( Rosenberg, 1993 ). In 1993 Loomis and colleagues produced an AR GPS-based system for helping the blind in the assisted navigation through adding spatial audio information ( Loomis et al., 1998 ). Always in the 1993 Julie Martin developed “Dancing in Cyberspace,” an AR theater in which actors interacted with virtual object in real time ( Cathy, 2011 ). Few years later, Feiner et al. (1997) developed the first Mobile AR System (MARS) able to add virtual information about touristic buildings ( Feiner et al., 1997 ). Since then, several applications have been developed: in Thomas et al. (2000) , created ARQuake, a mobile AR video game; in 2008 was created Wikitude that through the mobile camera, internet, and GPS could add information about the user’s environments ( Perry, 2008 ). In 2009 others AR applications, like AR Toolkit and SiteLens have been developed in order to add virtual information to the physical user’s surroundings. In 2011, Total Immersion developed D’Fusion, and AR system for designing projects ( Maurugeon, 2011 ). Finally, in 2013 and 2015, Google developed Google Glass and Google HoloLens, and their usability have begun to test in several field of application.

Virtual Reality Technologies

Technologically, the devices used in the virtual environments play an important role in the creation of successful virtual experiences. According to the literature, can be distinguished input and output devices ( Burdea et al., 1996 ; Burdea and Coiffet, 2003 ). Input devices are the ones that allow the user to communicate with the virtual environment, which can range from a simple joystick or keyboard to a glove allowing capturing finger movements or a tracker able to capture postures. More in detail, keyboard, mouse, trackball, and joystick represent the desktop input devices easy to use, which allow the user to launch continuous and discrete commands or movements to the environment. Other input devices can be represented by tracking devices as bend-sensing gloves that capture hand movements, postures and gestures, or pinch gloves that detect the fingers movements, and trackers able to follow the user’s movements in the physical world and translate them in the virtual environment.

On the contrary, the output devices allow the user to see, hear, smell, or touch everything that happens in the virtual environment. As mentioned above, among the visual devices can be found a wide range of possibilities, from the simplest or least immersive (monitor of a computer) to the most immersive one such as VR glasses or helmets or HMD or CAVE systems.

Furthermore, auditory, speakers, as well as haptic output devices are able to stimulate body senses providing a more real virtual experience. For example, haptic devices can stimulate the touch feeling and force models in the user.

Virtual Reality Applications

Since its appearance, VR has been used in different fields, as for gaming ( Zyda, 2005 ; Meldrum et al., 2012 ), military training ( Alexander et al., 2017 ), architectural design ( Song et al., 2017 ), education ( Englund et al., 2017 ), learning and social skills training ( Schmidt et al., 2017 ), simulations of surgical procedures ( Gallagher et al., 2005 ), assistance to the elderly or psychological treatments are other fields in which VR is bursting strongly ( Freeman et al., 2017 ; Neri et al., 2017 ). A recent and extensive review of Slater and Sanchez-Vives (2016) reported the main VR application evidences, including weakness and advantages, in several research areas, such as science, education, training, physical training, as well as social phenomena, moral behaviors, and could be used in other fields, like travel, meetings, collaboration, industry, news, and entertainment. Furthermore, another review published this year by Freeman et al. (2017) focused on VR in mental health, showing the efficacy of VR in assessing and treating different psychological disorders as anxiety, schizophrenia, depression, and eating disorders.

There are many possibilities that allow the use of VR as a stimulus, replacing real stimuli, recreating experiences, which in the real world would be impossible, with a high realism. This is why VR is widely used in research on new ways of applying psychological treatment or training, for example, to problems arising from phobias (agoraphobia, phobia to fly, etc.) ( Botella et al., 2017 ). Or, simply, it is used like improvement of the traditional systems of motor rehabilitation ( Llorens et al., 2014 ; Borrego et al., 2016 ), developing games that ameliorate the tasks. More in detail, in psychological treatment, Virtual Reality Exposure Therapy (VRET) has showed its efficacy, allowing to patients to gradually face fear stimuli or stressed situations in a safe environment where the psychological and physiological reactions can be controlled by the therapist ( Botella et al., 2017 ).

Augmented Reality Concept

Milgram and Kishino (1994) , conceptualized the Virtual-Reality Continuum that takes into consideration four systems: real environment, augmented reality (AR), augmented virtuality, and virtual environment. AR can be defined a newer technological system in which virtual objects are added to the real world in real-time during the user’s experience. Per Azuma et al. (2001) an AR system should: (1) combine real and virtual objects in a real environment; (2) run interactively and in real-time; (3) register real and virtual objects with each other. Furthermore, even if the AR experiences could seem different from VRs, the quality of AR experience could be considered similarly. Indeed, like in VR, feeling of presence, level of realism, and the degree of reality represent the main features that can be considered the indicators of the quality of AR experiences. Higher the experience is perceived as realistic, and there is congruence between the user’s expectation and the interaction inside the AR environments, higher would be the perception of “being there” physically, and at cognitive and emotional level. The feeling of presence, both in AR and VR environments, is important in acting behaviors like the real ones ( Botella et al., 2005 ; Juan et al., 2005 ; Bretón-López et al., 2010 ; Wrzesien et al., 2013 ).

Augmented Reality Technologies

Technologically, the AR systems, however various, present three common components, such as a geospatial datum for the virtual object, like a visual marker, a surface to project virtual elements to the user, and an adequate processing power for graphics, animation, and merging of images, like a pc and a monitor ( Carmigniani et al., 2011 ). To run, an AR system must also include a camera able to track the user movement for merging the virtual objects, and a visual display, like glasses through that the user can see the virtual objects overlaying to the physical world. To date, two-display systems exist, a video see-through (VST) and an optical see-though (OST) AR systems ( Botella et al., 2005 ; Juan et al., 2005 , 2007 ). The first one, disclosures virtual objects to the user by capturing the real objects/scenes with a camera and overlaying virtual objects, projecting them on a video or a monitor, while the second one, merges the virtual object on a transparent surface, like glasses, through the user see the added elements. The main difference between the two systems is the latency: an OST system could require more time to display the virtual objects than a VST system, generating a time lag between user’s action and performance and the detection of them by the system.

Augmented Reality Applications

Although AR is a more recent technology than VR, it has been investigated and used in several research areas such as architecture ( Lin and Hsu, 2017 ), maintenance ( Schwald and De Laval, 2003 ), entertainment ( Ozbek et al., 2004 ), education ( Nincarean et al., 2013 ; Bacca et al., 2014 ; Akçayır and Akçayır, 2017 ), medicine ( De Buck et al., 2005 ), and psychological treatments ( Juan et al., 2005 ; Botella et al., 2005 , 2010 ; Bretón-López et al., 2010 ; Wrzesien et al., 2011a , b , 2013 ; see the review Chicchi Giglioli et al., 2015 ). More in detail, in education several AR applications have been developed in the last few years showing the positive effects of this technology in supporting learning, such as an increased-on content understanding and memory preservation, as well as on learning motivation ( Radu, 2012 , 2014 ). For example, Ibáñez et al. (2014) developed a AR application on electromagnetism concepts’ learning, in which students could use AR batteries, magnets, cables on real superficies, and the system gave a real-time feedback to students about the correctness of the performance, improving in this way the academic success and motivation ( Di Serio et al., 2013 ). Deeply, AR system allows the possibility to learn visualizing and acting on composite phenomena that traditionally students study theoretically, without the possibility to see and test in real world ( Chien et al., 2010 ; Chen et al., 2011 ).

As well in psychological health, the number of research about AR is increasing, showing its efficacy above all in the treatment of psychological disorder (see the reviews Baus and Bouchard, 2014 ; Chicchi Giglioli et al., 2015 ). For example, in the treatment of anxiety disorders, like phobias, AR exposure therapy (ARET) showed its efficacy in one-session treatment, maintaining the positive impact in a follow-up at 1 or 3 month after. As VRET, ARET provides a safety and an ecological environment where any kind of stimulus is possible, allowing to keep control over the situation experienced by the patients, gradually generating situations of fear or stress. Indeed, in situations of fear, like the phobias for small animals, AR applications allow, in accordance with the patient’s anxiety, to gradually expose patient to fear animals, adding new animals during the session or enlarging their or increasing the speed. The various studies showed that AR is able, at the beginning of the session, to activate patient’s anxiety, for reducing after 1 h of exposition. After the session, patients even more than to better manage animal’s fear and anxiety, ware able to approach, interact, and kill real feared animals.

Materials and Methods

Data collection.

The input data for the analyses were retrieved from the scientific database Web of Science Core Collection ( Falagas et al., 2008 ) and the search terms used were “Virtual Reality” and “Augmented Reality” regarding papers published during the whole timespan covered.

Web of science core collection is composed of: Citation Indexes, Science Citation Index Expanded (SCI-EXPANDED) –1970-present, Social Sciences Citation Index (SSCI) –1970-present, Arts and Humanities Citation Index (A&HCI) –1975-present, Conference Proceedings Citation Index- Science (CPCI-S) –1990-present, Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) –1990-present, Book Citation Index– Science (BKCI-S) –2009-present, Book Citation Index– Social Sciences & Humanities (BKCI-SSH) –2009-present, Emerging Sources Citation Index (ESCI) –2015-present, Chemical Indexes, Current Chemical Reactions (CCR-EXPANDED) –2009-present (Includes Institut National de la Propriete Industrielle structure data back to 1840), Index Chemicus (IC) –2009-present.

The resultant dataset contained a total of 21,667 records for VR and 9,944 records for AR. The bibliographic record contained various fields, such as author, title, abstract, and all of the references (needed for the citation analysis). The research tool to visualize the networks was Cite space v.4.0.R5 SE (32 bit) ( Chen, 2006 ) under Java Runtime v.8 update 91 (build 1.8.0_91-b15). Statistical analyses were conducted using Stata MP-Parallel Edition, Release 14.0, StataCorp LP. Additional information can be found in Supplementary Data Sheet 1 .

The betweenness centrality of a node in a network measures the extent to which the node is part of paths that connect an arbitrary pair of nodes in the network ( Freeman, 1977 ; Brandes, 2001 ; Chen, 2006 ).

Structural metrics include betweenness centrality, modularity, and silhouette. Temporal and hybrid metrics include citation burstness and novelty. All the algorithms are detailed ( Chen et al., 2010 ).

The analysis of the literature on VR shows a complex panorama. At first sight, according to the document-type statistics from the Web of Science (WoS), proceedings papers were used extensively as outcomes of research, comprising almost 48% of the total (10,392 proceedings), with a similar number of articles on the subject amounting to about 47% of the total of 10, 199 articles. However, if we consider only the last 5 years (7,755 articles representing about 36% of the total), the situation changes with about 57% for articles (4,445) and about 33% for proceedings (2,578). Thus, it is clear that VR field has changed in areas other than at the technological level.

About the subject category, nodes and edges are computed as co-occurring subject categories from the Web of Science “Category” field in all the articles.

According to the subject category statistics from the WoS, computer science is the leading category, followed by engineering, and, together, they account for 15,341 articles, which make up about 71% of the total production. However, if we consider just the last 5 years, these categories reach only about 55%, with a total of 4,284 articles (Table 1 and Figure 1 ).

TABLE 1. Category statistics from the WoS for the entire period and the last 5 years.

FIGURE 1. Category from the WoS: network for the last 5 years.

The evidence is very interesting since it highlights that VR is doing very well as new technology with huge interest in hardware and software components. However, with respect to the past, we are witnessing increasing numbers of applications, especially in the medical area. In particular, note its inclusion in the top 10 list of rehabilitation and clinical neurology categories (about 10% of the total production in the last 5 years). It also is interesting that neuroscience and neurology, considered together, have shown an increase from about 12% to about 18.6% over the last 5 years. However, historic areas, such as automation and control systems, imaging science and photographic technology, and robotics, which had accounted for about 14.5% of the total articles ever produced were not even in the top 10 for the last 5 years, with each one accounting for less than 4%.

About the countries, nodes and edges are computed as networks of co-authors countries. Multiple occurrency of a country in the same paper are counted once.

The countries that were very involved in VR research have published for about 47% of the total (10,200 articles altogether). Of the 10,200 articles, the United States, China, England, and Germany published 4921, 2384, 1497, and 1398, respectively. The situation remains the same if we look at the articles published over the last 5 years. However, VR contributions also came from all over the globe, with Japan, Canada, Italy, France, Spain, South Korea, and Netherlands taking positions of prominence, as shown in Figure 2 .

FIGURE 2. Country network (node dimension represents centrality).

Network analysis was conducted to calculate and to represent the centrality index ( Freeman, 1977 ; Brandes, 2001 ), i.e., the dimension of the node in Figure 2 . The top-ranked country, with a centrality index of 0.26, was the United States (2011), and England was second, with a centrality index of 0.25. The third, fourth, and fifth countries were Germany, Italy, and Australia, with centrality indices of 0.15, 0.15, and 0.14, respectively.

About the Institutions, nodes and edges are computed as networks of co-authors Institutions (Figure 3 ).

FIGURE 3. Network of institutions: the dimensions of the nodes represent centrality.

The top-level institutions in VR were in the United States, where three universities were ranked as the top three in the world for published articles; these universities were the University of Illinois (159), the University of South California (147), and the University of Washington (146). The United States also had the eighth-ranked university, which was Iowa State University (116). The second country in the ranking was Canada, with the University of Toronto, which was ranked fifth with 125 articles and McGill University, ranked 10 th with 103 articles.

Other countries in the top-ten list were Netherlands, with the Delft University of Technology ranked fourth with 129 articles; Italy, with IRCCS Istituto Auxologico Italiano, ranked sixth (with the same number of publication of the institution ranked fifth) with 125 published articles; England, which was ranked seventh with 125 articles from the University of London’s Imperial College of Science, Technology, and Medicine; and China with 104 publications, with the Chinese Academy of Science, ranked ninth. Italy’s Istituto Auxologico Italiano, which was ranked fifth, was the only non-university institution ranked in the top-10 list for VR research (Figure 3 ).

About the Journals, nodes, and edges are computed as journal co-citation networks among each journals in the corresponding field.

The top-ranked Journals for citations in VR are Presence: Teleoperators & Virtual Environments with 2689 citations and CyberPsychology & Behavior (Cyberpsychol BEHAV) with 1884 citations; however, looking at the last 5 years, the former had increased the citations, but the latter had a far more significant increase, from about 70% to about 90%, i.e., an increase from 1029 to 1147.

Following the top two journals, IEEE Computer Graphics and Applications ( IEEE Comput Graph) and Advanced Health Telematics and Telemedicine ( St HEAL T) were both left out of the top-10 list based on the last 5 years. The data for the last 5 years also resulted in the inclusion of Experimental Brain Research ( Exp BRAIN RES) (625 citations), Archives of Physical Medicine and Rehabilitation ( Arch PHYS MED REHAB) (622 citations), and Plos ONE (619 citations) in the top-10 list of three journals, which highlighted the categories of rehabilitation and clinical neurology and neuroscience and neurology. Journal co-citation analysis is reported in Figure 4 , which clearly shows four distinct clusters.

FIGURE 4. Co-citation network of journals: the dimensions of the nodes represent centrality. Full list of official abbreviations of WoS journals can be found here: https://images.webofknowledge.com/images/help/WOS/A_abrvjt.html .

Network analysis was conducted to calculate and to represent the centrality index, i.e., the dimensions of the nodes in Figure 4 . The top-ranked item by centrality was Cyberpsychol BEHAV, with a centrality index of 0.29. The second-ranked item was Arch PHYS MED REHAB, with a centrality index of 0.23. The third was Behaviour Research and Therapy (Behav RES THER), with a centrality index of 0.15. The fourth was BRAIN, with a centrality index of 0.14. The fifth was Exp BRAIN RES, with a centrality index of 0.11.

Who’s Who in VR Research

Authors are the heart and brain of research, and their roles in a field are to define the past, present, and future of disciplines and to make significant breakthroughs to make new ideas arise (Figure 5 ).

FIGURE 5. Network of authors’ numbers of publications: the dimensions of the nodes represent the centrality index, and the dimensions of the characters represent the author’s rank.

Virtual reality research is very young and changing with time, but the top-10 authors in this field have made fundamentally significant contributions as pioneers in VR and taking it beyond a mere technological development. The purpose of the following highlights is not to rank researchers; rather, the purpose is to identify the most active researchers in order to understand where the field is going and how they plan for it to get there.

The top-ranked author is Riva G, with 180 publications. The second-ranked author is Rizzo A, with 101 publications. The third is Darzi A, with 97 publications. The forth is Aggarwal R, with 94 publications. The six authors following these three are Slater M, Alcaniz M, Botella C, Wiederhold BK, Kim SI, and Gutierrez-Maldonado J with 90, 90, 85, 75, 59, and 54 publications, respectively (Figure 6 ).

FIGURE 6. Authors’ co-citation network: the dimensions of the nodes represent centrality index, and the dimensions of the characters represent the author’s rank. The 10 authors that appear on the top-10 list are considered to be the pioneers of VR research.

Considering the last 5 years, the situation remains similar, with three new entries in the top-10 list, i.e., Muhlberger A, Cipresso P, and Ahmed K ranked 7th, 8th, and 10th, respectively.

The authors’ publications number network shows the most active authors in VR research. Another relevant analysis for our focus on VR research is to identify the most cited authors in the field.

For this purpose, the authors’ co-citation analysis highlights the authors in term of their impact on the literature considering the entire time span of the field ( White and Griffith, 1981 ; González-Teruel et al., 2015 ; Bu et al., 2016 ). The idea is to focus on the dynamic nature of the community of authors who contribute to the research.

Normally, authors with higher numbers of citations tend to be the scholars who drive the fundamental research and who make the most meaningful impacts on the evolution and development of the field. In the following, we identified the most-cited pioneers in the field of VR Research.

The top-ranked author by citation count is Gallagher (2001), with 694 citations. Second is Seymour (2004), with 668 citations. Third is Slater (1999), with 649 citations. Fourth is Grantcharov (2003), with 563 citations. Fifth is Riva (1999), with 546 citations. Sixth is Aggarwal (2006), with 505 citations. Seventh is Satava (1994), with 477 citations. Eighth is Witmer (2002), with 454 citations. Ninth is Rothbaum (1996), with 448 citations. Tenth is Cruz-neira (1995), with 416 citations.

Citation Network and Cluster Analyses for VR

Another analysis that can be used is the analysis of document co-citation, which allows us to focus on the highly-cited documents that generally are also the most influential in the domain ( Small, 1973 ; González-Teruel et al., 2015 ; Orosz et al., 2016 ).

The top-ranked article by citation counts is Seymour (2002) in Cluster #0, with 317 citations. The second article is Grantcharov (2004) in Cluster #0, with 286 citations. The third is Holden (2005) in Cluster #2, with 179 citations. The 4th is Gallagher et al. (2005) in Cluster #0, with 171 citations. The 5th is Ahlberg (2007) in Cluster #0, with 142 citations. The 6th is Parsons (2008) in Cluster #4, with 136 citations. The 7th is Powers (2008) in Cluster #4, with 134 citations. The 8th is Aggarwal (2007) in Cluster #0, with 121 citations. The 9th is Reznick (2006) in Cluster #0, with 121 citations. The 10th is Munz (2004) in Cluster #0, with 117 citations.

The network of document co-citations is visually complex (Figure 7 ) because it includes 1000s of articles and the links among them. However, this analysis is very important because can be used to identify the possible conglomerate of knowledge in the area, and this is essential for a deep understanding of the area. Thus, for this purpose, a cluster analysis was conducted ( Chen et al., 2010 ; González-Teruel et al., 2015 ; Klavans and Boyack, 2015 ). Figure 8 shows the clusters, which are identified with the two algorithms in Table 2 .

FIGURE 7. Network of document co-citations: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank, and the numbers represent the strengths of the links. It is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past VR research to the current research.

FIGURE 8. Document co-citation network by cluster: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank and the red writing reports the name of the cluster with a short description that was produced with the mutual information algorithm; the clusters are identified with colored polygons.

TABLE 2. Cluster ID and silhouettes as identified with two algorithms ( Chen et al., 2010 ).

The identified clusters highlight clear parts of the literature of VR research, making clear and visible the interdisciplinary nature of this field. However, the dynamics to identify the past, present, and future of VR research cannot be clear yet. We analysed the relationships between these clusters and the temporal dimensions of each article. The results are synthesized in Figure 9 . It is clear that cluster #0 (laparoscopic skill), cluster #2 (gaming and rehabilitation), cluster #4 (therapy), and cluster #14 (surgery) are the most popular areas of VR research. (See Figure 9 and Table 2 to identify the clusters.) From Figure 9 , it also is possible to identify the first phase of laparoscopic skill (cluster #6) and therapy (cluster #7). More generally, it is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past VR research to the current research.

FIGURE 9. Network of document co-citation: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank and the red writing on the right hand side reports the number of the cluster, such as in Table 2 , with a short description that was extracted accordingly.

We were able to identify the top 486 references that had the most citations by using burst citations algorithm. Citation burst is an indicator of a most active area of research. Citation burst is a detection of a burst event, which can last for multiple years as well as a single year. A citation burst provides evidence that a particular publication is associated with a surge of citations. The burst detection was based on Kleinberg’s algorithm ( Kleinberg, 2002 , 2003 ). The top-ranked document by bursts is Seymour (2002) in Cluster #0, with bursts of 88.93. The second is Grantcharov (2004) in Cluster #0, with bursts of 51.40. The third is Saposnik (2010) in Cluster #2, with bursts of 40.84. The fourth is Rothbaum (1995) in Cluster #7, with bursts of 38.94. The fifth is Holden (2005) in Cluster #2, with bursts of 37.52. The sixth is Scott (2000) in Cluster #0, with bursts of 33.39. The seventh is Saposnik (2011) in Cluster #2, with bursts of 33.33. The eighth is Burdea et al. (1996) in Cluster #3, with bursts of 32.42. The ninth is Burdea and Coiffet (2003) in Cluster #22, with bursts of 31.30. The 10th is Taffinder (1998) in Cluster #6, with bursts of 30.96 (Table 3 ).

TABLE 3. Cluster ID and references of burst article.

Citation Network and Cluster Analyses for AR

Looking at Augmented Reality scenario, the top ranked item by citation counts is Azuma (1997) in Cluster #0, with citation counts of 231. The second one is Azuma et al. (2001) in Cluster #0, with citation counts of 220. The third is Van Krevelen (2010) in Cluster #5, with citation counts of 207. The 4th is Lowe (2004) in Cluster #1, with citation counts of 157. The 5th is Wu (2013) in Cluster #4, with citation counts of 144. The 6th is Dunleavy (2009) in Cluster #4, with citation counts of 122. The 7th is Zhou (2008) in Cluster #5, with citation counts of 118. The 8th is Bay (2008) in Cluster #1, with citation counts of 117. The 9th is Newcombe (2011) in Cluster #1, with citation counts of 109. The 10th is Carmigniani et al. (2011) in Cluster #5, with citation counts of 104.

The network of document co-citations is visually complex (Figure 10 ) because it includes 1000s of articles and the links among them. However, this analysis is very important because can be used to identify the possible conglomerate of knowledge in the area, and this is essential for a deep understanding of the area. Thus, for this purpose, a cluster analysis was conducted ( Chen et al., 2010 ; González-Teruel et al., 2015 ; Klavans and Boyack, 2015 ). Figure 11 shows the clusters, which are identified with the two algorithms in Table 3 .

FIGURE 10. Network of document co-citations: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank, and the numbers represent the strengths of the links. It is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past AR research to the current research.

FIGURE 11. Document co-citation network by cluster: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank and the red writing reports the name of the cluster with a short description that was produced with the mutual information algorithm; the clusters are identified with colored polygons.

The identified clusters highlight clear parts of the literature of AR research, making clear and visible the interdisciplinary nature of this field. However, the dynamics to identify the past, present, and future of AR research cannot be clear yet. We analysed the relationships between these clusters and the temporal dimensions of each article. The results are synthesized in Figure 12 . It is clear that cluster #1 (tracking), cluster #4 (education), and cluster #5 (virtual city environment) are the current areas of AR research. (See Figure 12 and Table 3 to identify the clusters.) It is possible to identify four historical phases (colors: blue, green, yellow, and red) from the past AR research to the current research.

FIGURE 12. Network of document co-citation: the dimensions of the nodes represent centrality, the dimensions of the characters represent the rank of the article rank and the red writing on the right hand side reports the number of the cluster, such as in Table 2 , with a short description that was extracted accordingly.

We were able to identify the top 394 references that had the most citations by using burst citations algorithm. Citation burst is an indicator of a most active area of research. Citation burst is a detection of a burst event, which can last for multiple years as well as a single year. A citation burst provides evidence that a particular publication is associated with a surge of citations. The burst detection was based on Kleinberg’s algorithm ( Kleinberg, 2002 , 2003 ). The top ranked document by bursts is Azuma (1997) in Cluster #0, with bursts of 101.64. The second one is Azuma et al. (2001) in Cluster #0, with bursts of 84.23. The third is Lowe (2004) in Cluster #1, with bursts of 64.07. The 4th is Van Krevelen (2010) in Cluster #5, with bursts of 50.99. The 5th is Wu (2013) in Cluster #4, with bursts of 47.23. The 6th is Hartley (2000) in Cluster #0, with bursts of 37.71. The 7th is Dunleavy (2009) in Cluster #4, with bursts of 33.22. The 8th is Kato (1999) in Cluster #0, with bursts of 32.16. The 9th is Newcombe (2011) in Cluster #1, with bursts of 29.72. The 10th is Feiner (1993) in Cluster #8, with bursts of 29.46 (Table 4 ).

TABLE 4. Cluster ID and silhouettes as identified with two algorithms ( Chen et al., 2010 ).

Our findings have profound implications for two reasons. At first the present work highlighted the evolution and development of VR and AR research and provided a clear perspective based on solid data and computational analyses. Secondly our findings on VR made it profoundly clear that the clinical dimension is one of the most investigated ever and seems to increase in quantitative and qualitative aspects, but also include technological development and article in computer science, engineer, and allied sciences.

Figure 9 clarifies the past, present, and future of VR research. The outset of VR research brought a clearly-identifiable development in interfaces for children and medicine, routine use and behavioral-assessment, special effects, systems perspectives, and tutorials. This pioneering era evolved in the period that we can identify as the development era, because it was the period in which VR was used in experiments associated with new technological impulses. Not surprisingly, this was exactly concomitant with the new economy era in which significant investments were made in information technology, and it also was the era of the so-called ‘dot-com bubble’ in the late 1990s. The confluence of pioneering techniques into ergonomic studies within this development era was used to develop the first effective clinical systems for surgery, telemedicine, human spatial navigation, and the first phase of the development of therapy and laparoscopic skills. With the new millennium, VR research switched strongly toward what we can call the clinical-VR era, with its strong emphasis on rehabilitation, neurosurgery, and a new phase of therapy and laparoscopic skills. The number of applications and articles that have been published in the last 5 years are in line with the new technological development that we are experiencing at the hardware level, for example, with so many new, HMDs, and at the software level with an increasing number of independent programmers and VR communities.

Finally, Figure 12 identifies clusters of the literature of AR research, making clear and visible the interdisciplinary nature of this field. The dynamics to identify the past, present, and future of AR research cannot be clear yet, but analyzing the relationships between these clusters and the temporal dimensions of each article tracking, education, and virtual city environment are the current areas of AR research. AR is a new technology that is showing its efficacy in different research fields, and providing a novel way to gather behavioral data and support learning, training, and clinical treatments.

Looking at scientific literature conducted in the last few years, it might appear that most developments in VR and AR studies have focused on clinical aspects. However, the reality is more complex; thus, this perception should be clarified. Although researchers publish studies on the use of VR in clinical settings, each study depends on the technologies available. Industrial development in VR and AR changed a lot in the last 10 years. In the past, the development involved mainly hardware solutions while nowadays, the main efforts pertain to the software when developing virtual solutions. Hardware became a commodity that is often available at low cost. On the other hand, software needs to be customized each time, per each experiment, and this requires huge efforts in term of development. Researchers in AR and VR today need to be able to adapt software in their labs.

Virtual reality and AR developments in this new clinical era rely on computer science and vice versa. The future of VR and AR is becoming more technological than before, and each day, new solutions and products are coming to the market. Both from software and hardware perspectives, the future of AR and VR depends on huge innovations in all fields. The gap between the past and the future of AR and VR research is about the “realism” that was the key aspect in the past versus the “interaction” that is the key aspect now. First 30 years of VR and AR consisted of a continuous research on better resolution and improved perception. Now, researchers already achieved a great resolution and need to focus on making the VR as realistic as possible, which is not simple. In fact, a real experience implies a realistic interaction and not just great resolution. Interactions can be improved in infinite ways through new developments at hardware and software levels.

Interaction in AR and VR is going to be “embodied,” with implication for neuroscientists that are thinking about new solutions to be implemented into the current systems ( Blanke et al., 2015 ; Riva, 2018 ; Riva et al., 2018 ). For example, the use of hands with contactless device (i.e., without gloves) makes the interaction in virtual environments more natural. The Leap Motion device 1 allows one to use of hands in VR without the use of gloves or markers. This simple and low-cost device allows the VR users to interact with virtual objects and related environments in a naturalistic way. When technology is able to be transparent, users can experience increased sense of being in the virtual environments (the so-called sense of presence).

Other forms of interactions are possible and have been developing continuously. For example, tactile and haptic device able to provide a continuous feedback to the users, intensifying their experience also by adding components, such as the feeling of touch and the physical weight of virtual objects, by using force feedback. Another technology available at low cost that facilitates interaction is the motion tracking system, such as Microsoft Kinect, for example. Such technology allows one to track the users’ bodies, allowing them to interact with the virtual environments using body movements, gestures, and interactions. Most HMDs use an embedded system to track HMD position and rotation as well as controllers that are generally placed into the user’s hands. This tracking allows a great degree of interaction and improves the overall virtual experience.

A final emerging approach is the use of digital technologies to simulate not only the external world but also the internal bodily signals ( Azevedo et al., 2017 ; Riva et al., 2017 ): interoception, proprioception and vestibular input. For example, Riva et al. (2017) recently introduced the concept of “sonoception” ( www.sonoception.com ), a novel non-invasive technological paradigm based on wearable acoustic and vibrotactile transducers able to alter internal bodily signals. This approach allowed the development of an interoceptive stimulator that is both able to assess interoceptive time perception in clinical patients ( Di Lernia et al., 2018b ) and to enhance heart rate variability (the short-term vagally mediated component—rMSSD) through the modulation of the subjects’ parasympathetic system ( Di Lernia et al., 2018a ).

In this scenario, it is clear that the future of VR and AR research is not just in clinical applications, although the implications for the patients are huge. The continuous development of VR and AR technologies is the result of research in computer science, engineering, and allied sciences. The reasons for which from our analyses emerged a “clinical era” are threefold. First, all clinical research on VR and AR includes also technological developments, and new technological discoveries are being published in clinical or technological journals but with clinical samples as main subject. As noted in our research, main journals that publish numerous articles on technological developments tested with both healthy and patients include Presence: Teleoperators & Virtual Environments, Cyberpsychology & Behavior (Cyberpsychol BEHAV), and IEEE Computer Graphics and Applications (IEEE Comput Graph). It is clear that researchers in psychology, neuroscience, medicine, and behavioral sciences in general have been investigating whether the technological developments of VR and AR are effective for users, indicating that clinical behavioral research has been incorporating large parts of computer science and engineering. A second aspect to consider is the industrial development. In fact, once a new technology is envisioned and created it goes for a patent application. Once the patent is sent for registration the new technology may be made available for the market, and eventually for journal submission and publication. Moreover, most VR and AR research that that proposes the development of a technology moves directly from the presenting prototype to receiving the patent and introducing it to the market without publishing the findings in scientific paper. Hence, it is clear that if a new technology has been developed for industrial market or consumer, but not for clinical purpose, the research conducted to develop such technology may never be published in a scientific paper. Although our manuscript considered published researches, we have to acknowledge the existence of several researches that have not been published at all. The third reason for which our analyses highlighted a “clinical era” is that several articles on VR and AR have been considered within the Web of Knowledge database, that is our source of references. In this article, we referred to “research” as the one in the database considered. Of course, this is a limitation of our study, since there are several other databases that are of big value in the scientific community, such as IEEE Xplore Digital Library, ACM Digital Library, and many others. Generally, the most important articles in journals published in these databases are also included in the Web of Knowledge database; hence, we are convinced that our study considered the top-level publications in computer science or engineering. Accordingly, we believe that this limitation can be overcome by considering the large number of articles referenced in our research.

Considering all these aspects, it is clear that clinical applications, behavioral aspects, and technological developments in VR and AR research are parts of a more complex situation compared to the old platforms used before the huge diffusion of HMD and solutions. We think that this work might provide a clearer vision for stakeholders, providing evidence of the current research frontiers and the challenges that are expected in the future, highlighting all the connections and implications of the research in several fields, such as clinical, behavioral, industrial, entertainment, educational, and many others.

Author Contributions

PC and GR conceived the idea. PC made data extraction and the computational analyses and wrote the first draft of the article. IG revised the introduction adding important information for the article. PC, IG, MR, and GR revised the article and approved the last version of the article after important input to the article rationale.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer GC declared a shared affiliation, with no collaboration, with the authors PC and GR to the handling Editor at the time of the review.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2018.02086/full#supplementary-material

• ^ https://www.leapmotion.com/

Akçayır, M., and Akçayır, G. (2017). Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educ. Res. Rev. 20, 1–11. doi: 10.1016/j.edurev.2016.11.002

CrossRef Full Text | Google Scholar

Alexander, T., Westhoven, M., and Conradi, J. (2017). “Virtual environments for competency-oriented education and training,” in Advances in Human Factors, Business Management, Training and Education , (Berlin: Springer International Publishing), 23–29. doi: 10.1007/978-3-319-42070-7_3

Andersen, S. M., and Thorpe, J. S. (2009). An if–thEN theory of personality: significant others and the relational self. J. Res. Pers. 43, 163–170. doi: 10.1016/j.jrp.2008.12.040

Azevedo, R. T., Bennett, N., Bilicki, A., Hooper, J., Markopoulou, F., and Tsakiris, M. (2017). The calming effect of a new wearable device during the anticipation of public speech. Sci. Rep. 7:2285. doi: 10.1038/s41598-017-02274-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., and MacIntyre, B. (2001). Recent advances in augmented reality. IEEE Comp. Graph. Appl. 21, 34–47. doi: 10.1109/38.963459

Bacca, J., Baldiris, S., Fabregat, R., and Graf, S. (2014). Augmented reality trends in education: a systematic review of research and applications. J. Educ. Technol. Soc. 17, 133.

Google Scholar

Bailenson, J. N., Yee, N., Merget, D., and Schroeder, R. (2006). The effect of behavioral realism and form realism of real-time avatar faces on verbal disclosure, nonverbal disclosure, emotion recognition, and copresence in dyadic interaction. Presence 15, 359–372. doi: 10.1162/pres.15.4.359

Baños, R. M., Botella, C., Garcia-Palacios, A., Villa, H., Perpiñá, C., and Alcaniz, M. (2000). Presence and reality judgment in virtual environments: a unitary construct? Cyberpsychol. Behav. 3, 327–335. doi: 10.1089/10949310050078760

Baños, R., Botella, C., García-Palacios, A., Villa, H., Perpiñá, C., and Gallardo, M. (2009). Psychological variables and reality judgment in virtual environments: the roles of absorption and dissociation. Cyberpsychol. Behav. 2, 143–148. doi: 10.1089/cpb.1999.2.143

Baus, O., and Bouchard, S. (2014). Moving from virtual reality exposure-based therapy to augmented reality exposure-based therapy: a review. Front. Hum. Neurosci. 8:112. doi: 10.3389/fnhum.2014.00112

Biocca, F. (1997). The cyborg’s dilemma: progressive embodiment in virtual environments. J. Comput. Mediat. Commun. 3. doi: 10.1111/j.1083-6101.1997

Biocca, F., Harms, C., and Gregg, J. (2001). “The networked minds measure of social presence: pilot test of the factor structure and concurrent validity,” in 4th Annual International Workshop on Presence , Philadelphia, PA, 1–9.

Blanke, O., Slater, M., and Serino, A. (2015). Behavioral, neural, and computational principles of bodily self-consciousness. Neuron 88, 145–166. doi: 10.1016/j.neuron.2015.09.029

Bohil, C. J., Alicea, B., and Biocca, F. A. (2011). Virtual reality in neuroscience research and therapy. Nat. Rev. Neurosci. 12:752. doi: 10.1038/nrn3122

Borrego, A., Latorre, J., Llorens, R., Alcañiz, M., and Noé, E. (2016). Feasibility of a walking virtual reality system for rehabilitation: objective and subjective parameters. J. Neuroeng. Rehabil. 13:68. doi: 10.1186/s12984-016-0174-171

Botella, C., Bretón-López, J., Quero, S., Baños, R. M., and García-Palacios, A. (2010). Treating cockroach phobia with augmented reality. Behav. Ther. 41, 401–413. doi: 10.1016/j.beth.2009.07.002

Botella, C., Fernández-Álvarez, J., Guillén, V., García-Palacios, A., and Baños, R. (2017). Recent progress in virtual reality exposure therapy for phobias: a systematic review. Curr. Psychiatry Rep. 19:42. doi: 10.1007/s11920-017-0788-4

Botella, C. M., Juan, M. C., Baños, R. M., Alcañiz, M., Guillén, V., and Rey, B. (2005). Mixing realities? An application of augmented reality for the treatment of cockroach phobia. Cyberpsychol. Behav. 8, 162–171. doi: 10.1089/cpb.2005.8.162

Brandes, U. (2001). A faster algorithm for betweenness centrality. J. Math. Sociol. 25, 163–177. doi: 10.1080/0022250X.2001.9990249

Bretón-López, J., Quero, S., Botella, C., García-Palacios, A., Baños, R. M., and Alcañiz, M. (2010). An augmented reality system validation for the treatment of cockroach phobia. Cyberpsychol. Behav. Soc. Netw. 13, 705–710. doi: 10.1089/cyber.2009.0170

Brown, A., and Green, T. (2016). Virtual reality: low-cost tools and resources for the classroom. TechTrends 60, 517–519. doi: 10.1007/s11528-016-0102-z

Bu, Y., Liu, T. Y., and Huang, W. B. (2016). MACA: a modified author co-citation analysis method combined with general descriptive metadata of citations. Scientometrics 108, 143–166. doi: 10.1007/s11192-016-1959-5

Burdea, G., Richard, P., and Coiffet, P. (1996). Multimodal virtual reality: input-output devices, system integration, and human factors. Int. J. Hum. Compu. Interact. 8, 5–24. doi: 10.1080/10447319609526138

Burdea, G. C., and Coiffet, P. (2003). Virtual Reality Technology , Vol. 1, Hoboken, NJ: John Wiley & Sons.

Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., and Ivkovic, M. (2011). Augmented reality technologies, systems and applications. Multimed. Tools Appl. 51, 341–377. doi: 10.1007/s11042-010-0660-6

Castelvecchi, D. (2016). Low-cost headsets boost virtual reality’s lab appeal. Nature 533, 153–154. doi: 10.1038/533153a

Cathy (2011). The History of Augmented Reality. The Optical Vision Site. Available at: http://www.theopticalvisionsite.com/history-of-eyewear/the-history-of-augmented-reality/#.UelAUmeAOyA

Chen, C. (2006). CiteSpace II: detecting and visualizing emerging trends and transient patterns in scientific literature. J. Assoc. Inform. Sci. Technol. 57, 359–377. doi: 10.1002/asi.20317

Chen, C., Ibekwe-SanJuan, F., and Hou, J. (2010). The structure and dynamics of cocitation clusters: a multipleperspective cocitation analysis. J. Assoc. Inform. Sci. Technol. 61, 1386–1409. doi: 10.1002/jez.b.22741

Chen, Y. C., Chi, H. L., Hung, W. H., and Kang, S. C. (2011). Use of tangible and augmented reality models in engineering graphics courses. J. Prof. Issues Eng. Educ. Pract. 137, 267–276. doi: 10.1061/(ASCE)EI.1943-5541.0000078

Chicchi Giglioli, I. A., Pallavicini, F., Pedroli, E., Serino, S., and Riva, G. (2015). Augmented reality: a brand new challenge for the assessment and treatment of psychological disorders. Comput. Math. Methods Med. 2015:862942. doi: 10.1155/2015/862942

Chien, C. H., Chen, C. H., and Jeng, T. S. (2010). “An interactive augmented reality system for learning anatomy structure,” in Proceedings of the International Multiconference of Engineers and Computer Scientists , Vol. 1, (Hong Kong: International Association of Engineers), 17–19.

Choi, S., Jung, K., and Noh, S. D. (2015). Virtual reality applications in manufacturing industries: past research, present findings, and future directions. Concurr. Eng. 23, 40–63. doi: 10.1177/1063293X14568814

Cipresso, P. (2015). Modeling behavior dynamics using computational psychometrics within virtual worlds. Front. Psychol. 6:1725. doi: 10.3389/fpsyg.2015.01725

Cipresso, P., and Serino, S. (2014). Virtual Reality: Technologies, Medical Applications and Challenges. Hauppauge, NY: Nova Science Publishers, Inc.

Cipresso, P., Serino, S., and Riva, G. (2016). Psychometric assessment and behavioral experiments using a free virtual reality platform and computational science. BMC Med. Inform. Decis. Mak. 16:37. doi: 10.1186/s12911-016-0276-5

Cruz-Neira, C. (1993). “Virtual reality overview,” in SIGGRAPH 93 Course Notes 21st International Conference on Computer Graphics and Interactive Techniques, Orange County Convention Center , Orlando, FL.

De Buck, S., Maes, F., Ector, J., Bogaert, J., Dymarkowski, S., Heidbuchel, H., et al. (2005). An augmented reality system for patient-specific guidance of cardiac catheter ablation procedures. IEEE Trans. Med. Imaging 24, 1512–1524. doi: 10.1109/TMI.2005.857661

Di Lernia, D., Cipresso, P., Pedroli, E., and Riva, G. (2018a). Toward an embodied medicine: a portable device with programmable interoceptive stimulation for heart rate variability enhancement. Sensors (Basel) 18:2469. doi: 10.3390/s18082469

Di Lernia, D., Serino, S., Pezzulo, G., Pedroli, E., Cipresso, P., and Riva, G. (2018b). Feel the time. Time perception as a function of interoceptive processing. Front. Hum. Neurosci. 12:74. doi: 10.3389/fnhum.2018.00074

Di Serio,Á., Ibáñez, M. B., and Kloos, C. D. (2013). Impact of an augmented reality system on students’ motivation for a visual art course. Comput. Educ. 68, 586–596. doi: 10.1016/j.compedu.2012.03.002

Ebert, C. (2015). Looking into the future. IEEE Softw. 32, 92–97. doi: 10.1109/MS.2015.142

Englund, C., Olofsson, A. D., and Price, L. (2017). Teaching with technology in higher education: understanding conceptual change and development in practice. High. Educ. Res. Dev. 36, 73–87. doi: 10.1080/07294360.2016.1171300

Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., and Pappas, G. (2008). Comparison of pubmed, scopus, web of science, and Google scholar: strengths and weaknesses. FASEB J. 22, 338–342. doi: 10.1096/fj.07-9492LSF

Feiner, S., MacIntyre, B., Hollerer, T., and Webster, A. (1997). “A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment,” in Digest of Papers. First International Symposium on Wearable Computers , (Cambridge, MA: IEEE), 74–81. doi: 10.1109/ISWC.1997.629922

Freeman, D., Reeve, S., Robinson, A., Ehlers, A., Clark, D., Spanlang, B., et al. (2017). Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol. Med. 47, 2393–2400. doi: 10.1017/S003329171700040X

Freeman, L. C. (1977). A set of measures of centrality based on betweenness. Sociometry 40, 35–41. doi: 10.2307/3033543

Fuchs, H., and Bishop, G. (1992). Research Directions in Virtual Environments. Chapel Hill, NC: University of North Carolina at Chapel Hill.

Gallagher, A. G., Ritter, E. M., Champion, H., Higgins, G., Fried, M. P., Moses, G., et al. (2005). Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann. Surg. 241:364. doi: 10.1097/01.sla.0000151982.85062.80

Gigante, M. A. (1993). Virtual reality: definitions, history and applications. Virtual Real. Syst. 3–14. doi: 10.1016/B978-0-12-227748-1.50009-3

González-Teruel, A., González-Alcaide, G., Barrios, M., and Abad-García, M. F. (2015). Mapping recent information behavior research: an analysis of co-authorship and co-citation networks. Scientometrics 103, 687–705. doi: 10.1007/s11192-015-1548-z

Heeter, C. (1992). Being there: the subjective experience of presence. Presence 1, 262–271. doi: 10.1162/pres.1992.1.2.262

Heeter, C. (2000). Interactivity in the context of designed experiences. J. Interact. Adv. 1, 3–14. doi: 10.1080/15252019.2000.10722040

Heilig, M. (1962). Sensorama simulator. U.S. Patent No - 3, 870. Virginia: United States Patent and Trade Office.

Ibáñez, M. B., Di Serio,Á., Villarán, D., and Kloos, C. D. (2014). Experimenting with electromagnetism using augmented reality: impact on flow student experience and educational effectiveness. Comput. Educ. 71, 1–13. doi: 10.1016/j.compedu.2013.09.004

Juan, M. C., Alcañiz, M., Calatrava, J., Zaragozá, I., Baños, R., and Botella, C. (2007). “An optical see-through augmented reality system for the treatment of phobia to small animals,” in Virtual Reality, HCII 2007 Lecture Notes in Computer Science , Vol. 4563, ed. R. Schumaker (Berlin: Springer), 651–659.

Juan, M. C., Alcaniz, M., Monserrat, C., Botella, C., Baños, R. M., and Guerrero, B. (2005). Using augmented reality to treat phobias. IEEE Comput. Graph. Appl. 25, 31–37. doi: 10.1109/MCG.2005.143

Kim, G. J. (2005). A SWOT analysis of the field of virtual reality rehabilitation and therapy. Presence 14, 119–146. doi: 10.1162/1054746053967094

Klavans, R., and Boyack, K. W. (2015). Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge? J. Assoc. Inform. Sci. Technol. 68, 984–998. doi: 10.1002/asi.23734

Kleinberg, J. (2002). “Bursty and hierarchical structure in streams,” in Paper Presented at the Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2002; Edmonton , Alberta, NT. doi: 10.1145/775047.775061

Kleinberg, J. (2003). Bursty and hierarchical structure in streams. Data Min. Knowl. Discov. 7, 373–397. doi: 10.1023/A:1024940629314

Korolov, M. (2014). The real risks of virtual reality. Risk Manag. 61, 20–24.

Krueger, M. W., Gionfriddo, T., and Hinrichsen, K. (1985). “Videoplace—an artificial reality,” in Proceedings of the ACM SIGCHI Bulletin , Vol. 16, New York, NY: ACM, 35–40. doi: 10.1145/317456.317463

Lin, C. H., and Hsu, P. H. (2017). “Integrating procedural modelling process and immersive VR environment for architectural design education,” in MATEC Web of Conferences , Vol. 104, Les Ulis: EDP Sciences. doi: 10.1051/matecconf/201710403007

Llorens, R., Noé, E., Ferri, J., and Alcañiz, M. (2014). Virtual reality-based telerehabilitation program for balance recovery. A pilot study in hemiparetic individuals with acquired brain injury. Brain Inj. 28:169.

Lombard, M., and Ditton, T. (1997). At the heart of it all: the concept of presence. J. Comput. Mediat. Commun. 3. doi: 10.1111/j.1083-6101.1997.tb00072.x

Loomis, J. M., Blascovich, J. J., and Beall, A. C. (1999). Immersive virtual environment technology as a basic research tool in psychology. Behav. Res. Methods Instr. Comput. 31, 557–564. doi: 10.3758/BF03200735

Loomis, J. M., Golledge, R. G., and Klatzky, R. L. (1998). Navigation system for the blind: auditory display modes and guidance. Presence 7, 193–203. doi: 10.1162/105474698565677

Luckerson, V. (2014). Facebook Buying Oculus Virtual-Reality Company for $2 Billion. Available at: http://time.com/37842/facebook-oculus-rift

Maurugeon, G. (2011). New D’Fusion Supports iPhone4S and 3xDSMax 2012. Available at: http://www.t-immersion.com/blog/2011-12-07/augmented-reality-dfusion-iphone-3dsmax

Mazuryk, T., and Gervautz, M. (1996). Virtual Reality-History, Applications, Technology and Future. Vienna: Institute of Computer Graphics Vienna University of Technology.

Meldrum, D., Glennon, A., Herdman, S., Murray, D., and McConn-Walsh, R. (2012). Virtual reality rehabilitation of balance: assessment of the usability of the nintendo Wii ® fit plus. Disabil. Rehabil. 7, 205–210. doi: 10.3109/17483107.2011.616922

Milgram, P., and Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Trans. Inform. Syst. 77, 1321–1329.

Minderer, M., Harvey, C. D., Donato, F., and Moser, E. I. (2016). Neuroscience: virtual reality explored. Nature 533, 324–325. doi: 10.1038/nature17899

Neri, S. G., Cardoso, J. R., Cruz, L., Lima, R. M., de Oliveira, R. J., Iversen, M. D., et al. (2017). Do virtual reality games improve mobility skills and balance measurements in community-dwelling older adults? Systematic review and meta-analysis. Clin. Rehabil. 31, 1292–1304. doi: 10.1177/0269215517694677

Nincarean, D., Alia, M. B., Halim, N. D. A., and Rahman, M. H. A. (2013). Mobile augmented reality: the potential for education. Procedia Soc. Behav. Sci. 103, 657–664. doi: 10.1016/j.sbspro.2013.10.385

Orosz, K., Farkas, I. J., and Pollner, P. (2016). Quantifying the changing role of past publications. Scientometrics 108, 829–853. doi: 10.1007/s11192-016-1971-9

Ozbek, C. S., Giesler, B., and Dillmann, R. (2004). “Jedi training: playful evaluation of head-mounted augmented reality display systems,” in Proceedings of SPIE. The International Society for Optical Engineering , Vol. 5291, eds R. A. Norwood, M. Eich, and M. G. Kuzyk (Denver, CO), 454–463.

Perry, S. (2008). Wikitude: Android App with Augmented Reality: Mind Blow-Ing. Digital Lifestyles.

Radu, I. (2012). “Why should my students use AR? A comparative review of the educational impacts of augmented-reality,” in Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on , (IEEE), 313–314. doi: 10.1109/ISMAR.2012.6402590

Radu, I. (2014). Augmented reality in education: a meta-review and cross-media analysis. Pers. Ubiquitous Comput. 18, 1533–1543. doi: 10.1007/s00779-013-0747-y

Riva, G. (2018). The neuroscience of body memory: From the self through the space to the others. Cortex 104, 241–260. doi: 10.1016/j.cortex.2017.07.013

Riva, G., Gaggioli, A., Grassi, A., Raspelli, S., Cipresso, P., Pallavicini, F., et al. (2011). NeuroVR 2-A free virtual reality platform for the assessment and treatment in behavioral health care. Stud. Health Technol. Inform. 163, 493–495.

PubMed Abstract | Google Scholar

Riva, G., Serino, S., Di Lernia, D., Pavone, E. F., and Dakanalis, A. (2017). Embodied medicine: mens sana in corpore virtuale sano. Front. Hum. Neurosci. 11:120. doi: 10.3389/fnhum.2017.00120

Riva, G., Wiederhold, B. K., and Mantovani, F. (2018). Neuroscience of virtual reality: from virtual exposure to embodied medicine. Cyberpsychol. Behav. Soc. Netw. doi: 10.1089/cyber.2017.29099.gri [Epub ahead of print].

Rosenberg, L. (1993). “The use of virtual fixtures to enhance telemanipulation with time delay,” in Proceedings of the ASME Winter Anual Meeting on Advances in Robotics, Mechatronics, and Haptic Interfaces , Vol. 49, (New Orleans, LA), 29–36.

Schmidt, M., Beck, D., Glaser, N., and Schmidt, C. (2017). “A prototype immersive, multi-user 3D virtual learning environment for individuals with autism to learn social and life skills: a virtuoso DBR update,” in International Conference on Immersive Learning , Cham: Springer, 185–188. doi: 10.1007/978-3-319-60633-0_15

Schwald, B., and De Laval, B. (2003). An augmented reality system for training and assistance to maintenance in the industrial context. J. WSCG 11.

Serino, S., Cipresso, P., Morganti, F., and Riva, G. (2014). The role of egocentric and allocentric abilities in Alzheimer’s disease: a systematic review. Ageing Res. Rev. 16, 32–44. doi: 10.1016/j.arr.2014.04.004

Skalski, P., and Tamborini, R. (2007). The role of social presence in interactive agent-based persuasion. Media Psychol. 10, 385–413. doi: 10.1080/15213260701533102

Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364, 3549–3557. doi: 10.1098/rstb.2009.0138

Slater, M., and Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality. Front. Robot. AI 3:74. doi: 10.3389/frobt.2016.00074

Small, H. (1973). Co-citation in the scientific literature: a new measure of the relationship between two documents. J. Assoc. Inform. Sci. Technol. 24, 265–269. doi: 10.1002/asi.4630240406

Song, H., Chen, F., Peng, Q., Zhang, J., and Gu, P. (2017). Improvement of user experience using virtual reality in open-architecture product design. Proc. Inst. Mech. Eng. B J. Eng. Manufact. 232.

Sundar, S. S., Xu, Q., and Bellur, S. (2010). “Designing interactivity in media interfaces: a communications perspective,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , (Boston, MA: ACM), 2247–2256. doi: 10.1145/1753326.1753666

Sutherland, I. E. (1965). The Ultimate Display. Multimedia: From Wagner to Virtual Reality. New York, NY: Norton.

Sutherland, I. E. (1968). “A head-mounted three dimensional display,” in Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I , (ACM), 757–764. doi: 10.1145/1476589.1476686

Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi, P., Morris, M., et al. (2000). “ARQuake: an outdoor/indoor augmented reality first person application,” in Digest of Papers. Fourth International Symposium on Wearable Computers , (Atlanta, GA: IEEE), 139–146. doi: 10.1109/ISWC.2000.888480

Ware, C., Arthur, K., and Booth, K. S. (1993). “Fish tank virtual reality,” in Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems , (Amsterdam: ACM), 37–42. doi: 10.1145/169059.169066

Wexelblat, A. (ed.) (2014). Virtual Reality: Applications and Explorations. Cambridge, MA: Academic Press.

White, H. D., and Griffith, B. C. (1981). Author cocitation: a literature measure of intellectual structure. J. Assoc. Inform. Sci. Technol. 32, 163–171. doi: 10.1002/asi.4630320302

Wrzesien, M., Alcañiz, M., Botella, C., Burkhardt, J. M., Bretón-López, J., Ortega, M., et al. (2013). The therapeutic lamp: treating small-animal phobias. IEEE Comput. Graph. Appl. 33, 80–86. doi: 10.1109/MCG.2013.12

Wrzesien, M., Burkhardt, J. M., Alcañiz, M., and Botella, C. (2011a). How technology influences the therapeutic process: a comparative field evaluation of augmented reality and in vivo exposure therapy for phobia of small animals. Hum. Comput. Interact. 2011, 523–540.

Wrzesien, M., Burkhardt, J. M., Alcañiz Raya, M., and Botella, C. (2011b). “Mixing psychology and HCI in evaluation of augmented reality mental health technology,” in CHI’11 Extended Abstracts on Human Factors in Computing Systems , (Vancouver, BC: ACM), 2119–2124.

Zyda, M. (2005). From visual simulation to virtual reality to games. Computer 38, 25–32. doi: 10.1109/MC.2005.297

Keywords : virtual reality, augmented reality, quantitative psychology, measurement, psychometrics, scientometrics, computational psychometrics, mathematical psychology

Citation: Cipresso P, Giglioli IAC, Raya MA and Riva G (2018) The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature. Front. Psychol. 9:2086. doi: 10.3389/fpsyg.2018.02086

Received: 14 December 2017; Accepted: 10 October 2018; Published: 06 November 2018.

Reviewed by:

Copyright © 2018 Cipresso, Giglioli, Raya and Riva. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Pietro Cipresso, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Advertisement

A Meta-analysis of augmented reality programs for education and training

• Original Article
• Published: 16 August 2023
• Volume 27 , pages 2871–2894, ( 2023 )

Cite this article

• Matt C. Howard 1 &
• Maggie M. Davis 2  

497 Accesses

Explore all metrics

The application of augmented reality (AR) for education and training has grown dramatically in recent years, resulting in an expansive research domain within a relatively short amount of time. Two primary goals of the current article are to (a) summarize this literature by determining the overall effectiveness of AR programs relative to alternative comparisons and (b) assess the extent that AR program effectiveness is influenced by aspects of hardware, software, outcome, context, and methodology. A meta-analysis of over 250 studies supports that AR programs produce learning outcomes that are, on average, three-fifths of a standard deviation larger than alternative comparisons. Our results surprisingly show that AR programs using head-mounted displays produce significantly smaller effects than those using other output hardware (e.g., smartphones and tablets), and programs using image recognition are no more effective than those using alternative input methods (e.g., QR codes). We further find that most other aspects do not significantly influence observed program effectiveness; however, studies with younger participants produced significantly larger effects, and naturalistic studies produced significantly larger effects than laboratory studies. In our discussion, we utilize these findings to suggest promising theoretical perspectives for the study of AR, and we highlight methodological practices that can produce more accurate research moving forward. Thus, the current article summarizes research on AR education and training programs, identifies aspects that do and do not influence program efficacy, and provides several avenues for future research and practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Revealing the true potential and prospects of augmented reality in education

A Proposal of a Classification Scheme to a Survey of Augmented Reality for Education and Training

Emergent Individual Factors for AR Education and Training

To clarify the difference between marker detection and image recognition, we provide two examples. An AR application programmed to recognize specific pictures would be marker detection, because it recognizes static visual patterns. An example would be an AR program that presents the text, “COW,” whenever specific pictures of cows are seen but does not recognize pictures of cows more generally. An AR application programmed to recognize a category of things more broadly would be image recognition, because it recognizes real objects. An example would be an AR program that presents the text, “COW,” whenever any cow is seen and recognizes pictures of cows more generally. The main difference between marker detection and image recognition whether the AR program recognizes specific static images (marker detection) or things more broadly (image recognition).

Albert A, Hallowell MR, Kleiner B, Chen A, Golparvar-Fard M (2014) Enhancing construction hazard recognition with high-fidelity augmented virtuality. J Constr Eng Manag 140(7):04014024

Google Scholar  

Aromaa S, Väätänen A, Aaltonen I, Goriachev V, Helin K, Karjalainen J (2020) Awareness of the real-world environment when using augmented reality head-mounted display. Appl Ergon 88:103145

Au J, Sheehan E, Tsai N, Duncan GJ, Buschkuehl M, Jaeggi SM (2015) Improving fluid intelligence with training on working memory: a meta-analysis. Psychon Bull Rev 22(2):366–377

Aydoğdu F (2022) Augmented reality for preschool children: an experience with educational contents. Br J Edu Technol 53(2):326–348

Azuma RT (1997) A survey of AR. Presence Teleoper Virt Environ 6(4):355–385

Baker J, Wattie N, Schorer J (2015) Defining expertise: a taxonomy for researchers in skill acquisition and expertise. In Routledge Handbook of Sport Expertise. Routledge, pp 145–155

Baragash RS, Al-Samarraie H, Alzahrani AI, Alfarraj O (2020) Augmented reality in special education: a meta-analysis of single-subject design studies. Eur J Spec Needs Educ 35(3):382–397

Barmaki R, Yu K, Pearlman R, Shingles R, Bork F, Osgood GM, Navab N (2019) Enhancement of anatomical education using AR: an empirical study of body painting. Anat Sci Educ 12(6):599–609

Bedwell W, Pavlas D, Heyne K, Lazzara E, Salas E (2012) Toward a taxonomy linking game attributes to learning: an empirical study. Simul Gaming 43(6):729–760

Bell E, Bryman A, Harley B (2022) Business research methods. Oxford University Press

Bhandari S, Hallowell MR, Boven LV, Welker KM, Golparvar-Fard M, Gruber J (2020) Using augmented virtuality to examine how emotions influence construction-hazard identification, risk assessment, and safety decisions. J Constr Eng Manag 2:04019102

Bölek K, De Jong G, Henssen D (2021) The effectiveness of the use of augmented reality in anatomy education: a systematic review and meta-analysis. Sci Rep 11(1):1–10

Borenstein M, Hedges L, Higgins J, Rothstein H (2010) A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods 1(2):97–111

Borsci S, Lawson G, Broome S (2015) Empirical evidence, evaluation criteria and challenges for the effectiveness of virtual and mixed reality tools for training operators of car service maintenance. Comput Ind 67:17–26

Boxer P, Groves CL, Docherty M (2015) Video games do indeed influence children and adolescents’ aggression, prosocial behavior, and academic performance: a clearer reading of Ferguson (2015). Perspect Psychol Sci 10(5):671–673

Buchner J, Buntins K, Kerres M (2021) A systematic map of research characteristics in studies on augmented reality and cognitive load. Comput Educ Open 2:100036

Buchner J, Buntins K, Kerres M (2022) The impact of augmented reality on cognitive load and performance: a systematic review. J Comput Assist Learn 38(1):285–303

Cai Z, Fan X, Du J (2017) Gender and attitudes toward technology use: a meta-analysis. Comput Educ 105:1–13

Challenor J, Ma M (2019) A review of augmented reality applications for history education and heritage visualisation. Multim Technol Interact 3(2):39

Chen Y, Wang Q, Chen H, Song X, Tang H, Tian M (2019) An overview of augmented reality technology. J Phys Conf Ser 1237:022082

Cheng K, Tsai C (2013) Affordances of augmented reality in science learning: suggestions for future research. J Sci Educ Technol 22(4):449–462

Cheng LJ, Kumar PA, Wong SN, Lau Y (2020a) Technology-delivered psychotherapeutic interventions in improving depressive symptoms among people with HIV/AIDS: a systematic review and meta-analysis of randomised controlled trials. AIDS Behav 24(6):1663–1675

Cheng Q, Zhang S, Bo S, Chen D, Zhang H (2020b) Augmented reality dynamic image recognition technology based on deep learning algorithm. IEEE Access 8:137370–137384

Cheung M (2015) Meta-analysis: a structural equation modeling approach. John Wiley & Sons

MATH   Google Scholar  

Cifuentes SC, García SG, Andrés-Sebastiá MP, Camba JD, Contero M (2016) Augmented reality experiences in therapeutic pedagogy: a study with special needs students. In 2016 IEEE 16th international conference on advanced learning technologies (ICALT), pp 431–435. IEEE.

Cohen NJ, Eichenbaum H, Deacedo BS, Corkin S (1985) Different memory systems underlying acquisition of procedural and declarative knowledge. Ann N Y Acad Sci 444:54–71

Dalton DR, Aguinis H, Dalton CM, Bosco FA, Pierce CA (2012) Revisiting the file drawer problem in meta-analysis: an assessment of published and nonpublished correlation matrices. Pers Psychol 65(2):221–249

Das P, Zhu MO, McLaughlin L, Bilgrami Z, Milanaik RL (2017) Augmented reality video games: new possibilities and implications for children and adolescents. Multim Technol Inter 1(2):8

Ferguson CJ (2015) Pay no attention to that data behind the curtain: on angry birds, happy children, scholarly squabbles, publication bias, and why betas rule metas. Perspect Psychol Sci 10(5):683–691

Foo CC, Cheung B, Chu KM (2021) A comparative study regarding distance learning and the conventional face-to-face approach conducted problem-based learning tutorial during the COVID-19 pandemic. BMC Med Educ 21(1):1–6

Garzón J (2021) An overview of twenty-five years of augmented reality in education. Multim Technol Inter 5(7):37

Garzón J, Acevedo J (2019) Meta-analysis of the impact of augmented reality on students’ learning gains. Educ Res Rev 27:244–260

Garzón J, Pavón J, Baldiris S (2019) Systematic review and meta-analysis of augmented reality in educational settings. Virt Real 23(4):447–459

Garzón J, Baldiris S, Gutiérrez J, Pavón J (2020) How do pedagogical approaches affect the impact of augmented reality on education? A meta-analysis and research synthesis. Educ Res Rev 31:100334

Gavish N, Gutierrez T, Webel S, Rodriguez J, Tecchia F (2011) Design guidelines for the development of virtual reality and augmented reality training systems for maintenance and assembly tasks. BIO Web Conf 1:29

Gavish N (2019) The dark side of using augmented reality (AR) training systems in industry. In: Systems engineering in the fourth industrial revolution, pp 191–201

Gerup J, Soerensen CB, Dieckmann P (2020) Augmented reality and mixed reality for healthcare education beyond surgery: an integrative review. Int J Med Educ 11:1

Goh GS, Lohre R, Parvizi J, Goel DP (2021) Virtual and augmented reality for surgical training and simulation in knee arthroplasty. Arch Orthop Trauma Surg 141:2303–2312

Gonzalez DAZ, Richards D, Bilgin AA (2021) Making it real: a study of augmented virtuality on presence and enhanced benefits of study stress reduction sessions. Int J Hum Comput Stud 147:102579

Goodhue DL, Thompson RL (1995) Task-technology fit and individual performance. MIS Quar 1:213–236

Grassini S, Laumann K (2020) Are modern head-mounted displays sexist? A systematic review on gender differences in HMD-mediated virtual reality. Front Psychol 11:1604

Gray JR, Chabris CF, Braver TS (2003) Neural mechanisms of general fluid intelligence. Nat Neurosci 6(3):316–322

Greene JA, Duke RF, Freed R, Dragnić-Cindrić D, Cartiff BM (2022) Effects of an ego-depletion intervention upon online learning. Comput Educ 177:104362

Gu W, Shah K, Knopf J, Navab N, Unberath M (2021) Feasibility of image-based augmented reality guidance of total shoulder arthroplasty using Microsoft HoloLens 1. Comput Methods Biomech Biomed Eng Imag Visualiz 9(3):261–270

Guinet AL, Bouyer G, Otmane S, Desailly E (2021) Validity of HoloLens augmented reality head mounted display for measuring gait parameters in healthy adults and children with cerebral palsy. Sensors 21(8):2697

Hair J, Page M, Brunsveld N (2019) Essentials of business research methods. Routledge

Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with Microsoft Kinect sensor: a review. IEEE Trans Cybern 43(5):1318–1334

Harrison TL, Shipstead Z, Hicks KL, Hambrick DZ, Redick TS, Engle RW (2013) Working memory training may increase working memory capacity but not fluid intelligence. Psychol Sci 24(12):2409–2419

Hedges LV, Vevea JL (1998) Fixed-and random-effects models in meta-analysis. Psychol Methods 3(4):486

Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, Welch V (eds) (2019) Cochrane handbook for systematic reviews of interventions. John Wiley & Sons

Howard MC, Davis MM (2022) A meta-analysis and systematic literature review of mixed reality rehabilitation programs: investigating design characteristics of augmented reality and augmented virtuality. Comput Human Beh 130:107197

Howard MC, Lee J (2020) Pre-training interventions to counteract seductive details in virtual reality training programs. Hum Resour Dev Q 31(1):13–29

Howard MC, Rose JC (2019) Refining and extending task–technology fit theory: creation of two task–technology fit scales and empirical clarification of the construct. Information & Manag 56(6):103134

Howard MC, Van Zandt EC (2021) A meta-analysis of the virtual reality problem: unequal effects of virtual reality sickness across individual differences. Virtual Real 25(4):1221–1246

Howard MC, Gutworth MB, Jacobs RR (2021) A meta-analysis of virtual reality training programs. Comput Hum Behav 121:106808

Huang TC, Chen CC, Chou YW (2016) Animating eco-education: to see, feel, and discover in an AR-based experiential learning environment. Comput Educ 96:72–82

İbili E, Çat M, Resnyansky D, Şahin S, Billinghurst M (2020) An assessment of geometry teaching supported with augmented reality teaching materials to enhance students’ 3D geometry thinking skills. Int J Math Educ Sci Technol 51(2):224–246

Innocente C, Ulrich L, Moos S, Vezzetti E (2022) AR: mapping methods and tools for enhancing the human role in healthcare HMI. Appl Sci 12(9):4295

Itoh Y, Langlotz T, Sutton J, Plopski A (2021) Towards indistinguishable augmented reality: a survey on optical see-through head-mounted displays. ACM Comput Surv 54(6):1–36

Ivan ME, Eichberg DG, Di L, Shah AH, Luther EM, Lu VM, Urakov TM (2021) Augmented reality head-mounted display–based incision planning in cranial neurosurgery: a prospective pilot study. Neurosurg Focus 51(2):E3

Jak S (2015) Meta-analytic structural equation modelling. Springer, Dordrecht, pp 1–88

Kaplan AD, Cruit J, Endsley M, Beers SM, Sawyer BD, Hancock PA (2021) The effects of virtual reality, AR, and mixed reality as training enhancement methods: a meta-analysis. Hum Factors 63(4):706–726

Kaufeld M, Mundt M, Forst S, Hecht H (2022) Optical see-through augmented reality can induce severe motion sickness. Displays 74:102283

Khamzina M, Parab KV, An R, Bullard T, Grigsby-Toussaint DS (2020) Impact of Pokémon Go on physical activity: a systematic review and meta-analysis. Am J Prev Med 58(2):270–282

Kirkpatrick DL (1996) Great ideas revisited: revisiting Kirkpatrick’s four-level model. Train Dev 50(1):54–57

Kolb DA (2014) Experiential learning: experience as the source of learning and development. FT press, London

Kolb DA, Boyatzis RE, Mainemelis C (2014). Experiential learning theory: previous research and new directions. In: Perspectives on thinking, learning, and cognitive styles. Routledge, pp 227–248

Kozlowski SWJ, DeShon RP (2004) A psychological fidelity approach to simulation-based training: theory, research, and principles. In: Salas E, Elliott LR, Schflett SG, Coovert MD (eds) Scaled worlds: development, validation, and applications. Ashgate Publishing, Burlington, pp 75–99

Lampropoulos G, Keramopoulos E, Diamantaras K, Evangelidis G (2022) Augmented reality and gamification in education: a systematic literature review of research, applications, and empirical studies. Appl Sci 12(13):6809

Landers R, Auer E, Collmus A, Armstrong M (2018) Gamification science, its history and future: definitions and a research agenda. Simul Gaming 49(3):315–337

Lauer L, Altmeyer K, Malone S, Barz M, Brünken R, Sonntag D, Peschel M (2021) Investigating the usability of a head-mounted display augmented reality device in elementary school children. Sensors 21(19):6623

Le Y, Chen Z, Liu S, Pang W, Deng C (2021) Investigating the effectiveness of emotional design principle to attenuate ego depletion effect. Comput Educ 174:104311

Lee K (2012) Augmented reality in education and training. TechTrends 56(2):13–21

Lee IJ, Chen CH, Chang KP (2016) Augmented reality technology combined with three-dimensional holography to train the mental rotation ability of older adults. Comput Hum Behav 65:488–500

Liu D, Macchiarella ND, Vincenzi DA (2008) Simulation fidelity. In: Human factors in simulation and training, pp 61–73

Liu Z, Lan G, Stojkovic J, Zhang Y, Joe-Wong C, Gorlatova M (2020) Collabar: edge-assisted collaborative image recognition for mobile AR. In: 2020 19th ACM/IEEE international conference on information processing in sensor networks (IPSN), pp 301–312. IEEE.

Mao CC, Chen CH (2021) Augmented reality of 3D content application in common operational picture training system for army. Int J Human Comput Inter 37(20):1899–1915

Marek MW, Chew CS, Wu WCV (2021) Teacher experiences in converting classes to distance learning in the COVID-19 pandemic. Int J Distance Educ Technol 19(1):89–109

MarketsandMarkets (2022) Augmented and virtual reality in education market. Retrieved from: https://www.marketsandmarkets.com/Market-Reports/virtual-classroom-market-203811025.html .

Mesmer-Magnus J, Viswesvaran C (2010) The role of pre-training interventions in learning: a meta-analysis and integrative review. Hum Resour Manag Rev 20(4):261–282

Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 77(12):1321–1329

Milman NB (2018) Defining and conceptualizing mixed reality, augmented reality, and virtual reality. Distance Learn 15(2):55–58

Moesl B, Schaffernak H, Vorraber W, Braunstingl R, Herrele T, Koglbauer IV (2021) A research agenda for implementing augmented reality in ab initio pilot training. Aviat Psychol Appl Human Fact

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Stewart LA (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 4(1):1–9

Moorhouse N et al (2019) An experiential view to children learning in museums with AR. Museum Manag Curat 34(4):402–418

Moreno R (2006) Does the modality principle hold for different media? A test of the method-affects-learning hypothesis. J Comput Assist Learn 22(3):149–158

MathSciNet   Google Scholar  

Morice AH, Rakotomamonjy T, Serres JR, Ruffier F (2021) Ecological design of augmentation improves helicopter ship landing maneuvers: an approach in augmented virtuality. PLoS ONE 16(8):e0255779

Moro C, Birt J, Stromberga Z, Phelps C, Clark J, Glasziou P, Scott AM (2021) Virtual and augmented reality enhancements to medical and science student physiology and anatomy test performance: a systematic review and meta-analysis. Anat Sci Educ 14(3):368–376

Mubin O, Alnajjar F, Jishtu N, Alsinglawi B, Al Mahmud A (2019) Exoskeletons with virtual reality, augmented reality, and gamification for stroke patients’ rehabilitation: systematic review. JMIR Rehab Assist Technol 6(2):e12010

Newbutt N, Bradley R, Conley I (2020) Using virtual reality head-mounted displays in schools with autistic children: views, experiences, and future directions. Cyberpsychol Behav Soc Netw 23(1):23–33

Noll C, von Jan U, Raap U, Albrecht UV (2017) Mobile augmented reality as a feature for self-oriented, blended learning in medicine: randomized controlled trial. JMIR Mhealth Uhealth 5(9):e7943

Noreikis M, Savela N, Kaakinen M, Xiao Y, Oksanen A (2019) Effects of gamified augmented reality in public spaces. IEEE Access 7:148108–148118

Ozdemir M, Sahin C, Arcagok S, Demir MK (2018) The effect of augmented reality applications in the learning process: a meta-analysis study. Eurasian J Educ Res 18(74):165–186

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Moher D (2021a) Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. J Clin Epidemiol 134:103–112

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Moher D (2021b) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev 10(1):1–11

Park S, Bokijonov S, Choi Y (2021) Review of Microsoft HoloLens applications over the past five years. Appl Sci 11(16):7259

Patall E, Cooper H, Robinson J (2008) The effects of choice on intrinsic motivation and related outcomes: a meta-analysis of research findings. Psychol Bull 134(2):270

Pettijohn KA, Peltier C, Lukos JR, Norris JN, Biggs AT (2020) Virtual and augmented reality in a simulated naval engagement: preliminary comparisons of simulator sickness and human performance. Appl Ergon 89:103200

Plass JL, Kaplan U (2016) Emotional design in digital media for learning. In: Emotions, technology, design, and learning. Academic Press, pp 131–161

Rese A, Baier D, Geyer-Schulz A, Schreiber S (2017) How augmented reality apps are accepted by consumers: a comparative analysis using scales and opinions. Technol Forecast Soc Chang 124:306–319

Rokhsaritalemi S, Sadeghi-Niaraki A, Choi SM (2020) A review on mixed reality: Current trends, challenges and prospects. Appl Sci 10(2):636

Rosen Y, Salomon G (2007) The differential learning achievements of constructivist technology-intensive learning environments as compared with traditional ones: a meta-analysis. J Educ Comput Res 36(1):1–14

Rourke L, Schmidt M, Garga N (2010) Theory-based research of high fidelity simulation use in nursing education: a review of the literature. Int J Nurs Educ Schol 7:1

Ruiz-Ariza A, Casuso RA, Suarez-Manzano S, Martínez-López EJ (2018) Effect of augmented reality game Pokémon GO on cognitive performance and emotional intelligence in adolescent young. Comput Educ 116:49–63

Santos MEC, Taketomi T, Yamamoto G, Rodrigo M, Mercedes T, Sandor C, Kato H (2016) Augmented reality as multimedia: the case for situated vocabulary learning. Res Pract Technol Enhanc Learn 11(1):1–23

Schneider S, Beege M, Nebel S, Schnaubert L, Rey GD (2021) The cognitive-affective-social theory of learning in digital environments (CASTLE). Educ Psychol Rev, 1–38

Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, Stewart LA (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 349:1–9

Spellman BA (2012) Introduction to the special section: Data, data, everywhere… especially in my file drawer. Perspect Psychol Sci 7(1):58

Sternberg RJ (2008) Increasing fluid intelligence is possible after all. Proc Natl Acad Sci 105(19):6791–6792

Stevens J, Kincaid P, Sottilare R (2015) Visual modality research in virtual and mixed reality simulation. J Defense Mod Simul 12(4):519–537

Stretton T, Cochrane T, Narayan V (2018) Exploring mobile mixed reality in healthcare higher education: a systematic review. Res Learn Technol 26:2131–2131

Sundararajan N, Adesope O (2020) Keep it coherent: a meta-analysis of the seductive details effect. Educ Psychol Rev 32(3):707–734

Suresh D, Aydin A, James S, Ahmed K, Dasgupta P (2022) The role of augmented reality in surgical training: a systematic review. Surg Innov

Tekedere H, Göke H (2016) Examining the effectiveness of augmented reality applications in education: a meta-analysis. Int J Environ Sci Educ 11(16):9469–9481

Ten Berge T, Van Hezewijk R (1999) Procedural and declarative knowledge: an evolutionary perspective. Theory Psychol 9(5):605–624

Tepper O, Rudy H, Lefkowitz A, Weimer K, Marks S, Stern C, Garfein E (2017) Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room. Plast Reconstr Surg 140(5):1066–1070

Tuli N, Mantri A, Sharma S (2022) Augmented reality n education: a systematic study on technical and usability issues. Int J Comput Aided Eng Technol 17(2):164–183

Uttl B (2008) Transparent meta-analysis of prospective memory and aging. PLoS ONE 3(2):e1568

Van der Kooij K, Van Dijsseldonk R, Van Veen M, Steenbrink F, De Weerd C, Overvliet KE (2019) Gamification as a sustainable source of enjoyment during balance and gait exercises. Front Psychol 10:294

Vellingiri S, McMahan RP, Johnson V, Prabhakaran B (2022) An augmented virtuality system facilitating learning through nature walk. Multim Tools Appl, 1–12

Vergel RS, Tena PM, Yrurzum SC, Cruz-Neira C (2020) A comparative evaluation of a virtual reality table and a HoloLens-based augmented reality system for anatomy training. IEEE Trans Human Mach Syst 50(4):337–348

Viechtbauer W, Cheung MWL (2010) Outlier and influence diagnostics for meta-analysis. Res Synth Methods 1(2):112–125

Viglialoro RM, Condino S, Turini G, Carbone M, Ferrari V, Gesi M (2021) Augmented reality, mixed reality, and hybrid approach in healthcare simulation: a systematic review. Appl Sci 11(5):2338

Wang X, Dunston PS (2007) Design, strategies, and issues towards an augmented reality-based construction training platform. J Inf Technol Constr 12(25):363–380

Webel S, Bockholt U, Engelke T, Gavish N, Olbrich M, Preusche C (2013) An augmented reality training platform for assembly and maintenance skills. Robot Auton Syst 61(4):398–403

Xiao M, Feng Z, Yang X, Xu T, Guo Q (2020) Multimodal interaction design and application in augmented reality for chemical experiment. Virt Real Intel Hardw 2(4):291–304

Xu W, Qin X, Li X, Chen H, Frank M, Rutherford A et al (2021) Developing China’s workforce skill taxonomy reveals extent of labor market polarization. Human Soc Sci Commun 8(1):1–10

Xu W, Liang HN, He A, Wang Z (2019) Pointing and selection methods for text entry in augmented reality head mounted displays. In 2019 IEEE international symposium on mixed and AR, 279–288.

Yu H, Zhou Z, Lei X, Liu H, Fan G, He S (2019) Mixed reality− based preoperative planning for training of percutaneous transforaminal endoscopic discectomy: a feasibility study. World Neurosurg 129:e767–e775

Zackoff M, Cruse B, Sahay RD, Fei L, Saupe J, Schwartz J et al (2021) Development and implementation of augmented reality enhanced high-fidelity simulation for recognition of patient decompensation. Simul Healthcare 16(3):221–230

Zhang Z (2012) Microsoft Kinect sensor and its effect. IEEE Multim 19(2):4–10

Zhou Z, Jiang S, Yang Z, Zhou L (2019) Personalized planning and training system for brachytherapy based on virtual reality. Virtual Real 23:347–361

Download references

Author information

Authors and affiliations.

Mitchell College of Business, The University of South Alabama, Mobile, AL, USA

Matt C. Howard

College of Business, University of West Florida, Pensacola, FL, USA

Maggie M. Davis

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Matt C. Howard .

Ethics declarations

Conflict of interest.

The authors have no expressed conflict of interest.

Human or animal rights

The current article reports a meta-analysis, and it does not involve the collection of primary data from human or animal participants.

Informed consent

Because no primary data were collected for the current article, no informed consent was obtained.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (CMA 1532 KB)

Supplementary file2 (xlsx 176 kb), supplementary file3 (docx 63 kb), supplementary file4 (docx 45 kb), supplementary file5 (docx 39 kb).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Howard, M.C., Davis, M.M. A Meta-analysis of augmented reality programs for education and training. Virtual Reality 27 , 2871–2894 (2023). https://doi.org/10.1007/s10055-023-00844-6

Download citation

Received : 05 January 2023

Accepted : 28 July 2023

Published : 16 August 2023

Issue Date : December 2023

DOI : https://doi.org/10.1007/s10055-023-00844-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Augmented reality
• Mixed reality
• Meta-analysis
• Find a journal
• Publish with us
• Track your research

University of Minnesota Twin Cities

University digital conservancy, exploring student engagement in an augmented reality game, view/download file, persistent link to this item, journal title, journal issn, volume title, published date, description, collections, series/report number, funding information, isbn identifier, doi identifier, previously published citation, suggested citation.

Content distributed via the University Digital Conservancy may be subject to additional license and use restrictions applied by the depositor. By using these files, users agree to the Terms of Use . Materials in the UDC may contain content that is disturbing and/or harmful. For more information, please see our statement on harmful content in digital repositories .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 26 June 2023

The impact of augmented reality on student attitudes, motivation, and learning achievements—a meta-analysis (2016–2023)

• Wenwen Cao 1 &
• Zhonggen Yu   ORCID: orcid.org/0000-0002-3873-980X 2  

Humanities and Social Sciences Communications volume  10 , Article number:  352 ( 2023 ) Cite this article

6863 Accesses

11 Citations

Metrics details

• Science, technology and society

In light of the COVID-19 pandemic, a significant number of students have been compelled to remain at home while receiving education supported by augmented reality (AR) technologies. To determine the impact of AR technologies on educational outcomes, the present study undertook a meta-analysis utilizing Stata/MP 14.0. The study found that the attitudes of learners towards AR-assisted education were more positive, and their learning achievements were significantly higher compared to those who did not use AR technologies. However, there was no significant difference in motivation levels between the AR-assisted and non-AR-assisted educational models. The researchers explored several reasons for this result, but they could not identify any clear explanation. Future studies could take into account other factors that might affect education outcomes such as learning styles and learner personality. Doing so could shed more light on the impact of AR technologies on education.

Similar content being viewed by others

What colour are your eyes? Teaching the genetics of eye colour & colour vision. Edridge Green Lecture RCOphth Annual Congress Glasgow May 2019

Impact of artificial intelligence on human loss in decision making, laziness and safety in education

The impact of artificial intelligence on employment: the role of virtual agglomeration

Introduction.

Since the emergence of the COVID-19 pandemic, many students have been compelled to receive education from home with the assistance of augmented reality (AR) technologies (Saleem et al., 2021 ). Given the rising popularity of AR technologies in the field of education (Tezer et al., 2019 ), a multitude of studies have conducted meta-analyses to investigate their effectiveness, particularly under the COVID-19 pandemic context (e.g., Selek and Kiymaz, 2020 ; Bork et al., 2020 ; Gargrish et al., 2021 ; Gonzalez et al., 2020 ). One recent meta-analysis found that AR technologies could have a positive impact on learning outcomes when users’ spatial abilities were taken into account (Bölek et al., 2021 ). While medium-sized effects were often observed in terms of learning gains resulting from the use of AR (Garzón and Acevedo, 2019 ), the results may have been influenced by the exclusion of studies with insufficient data. Additionally, when applied in collaborative learning, AR technologies could have a major influence on learning outcomes, although the results were limited to the pedagogical methods utilized in the included sample (Garzón et al., 2020 ).

The field of education has witnessed a rapid surge in the popularity of augmented reality (AR), which has the potential to greatly enhance learning experiences (Garzón et al., 2019 ). However, the study conducted by Garzón et al. ( 2019 ) neglected to define the specific features of AR that can conveniently assist and improve learning achievements. When compared to traditional learning methods, AR-assisted learning has demonstrated a considerable improvement in learning achievements, and the efficacy of various AR applications in education has shown no significant differences (Ozdemir et al., 2018 ). It is important to note, however, that the sample size in Ozdemir et al.’s study was restricted to only 16 participants and was limited to the Social Sciences Citation Index, resulting in a possible sample bias that could impede the reliability of their results. Learner attitudes toward and learning achievements in AR-assisted education may need further examination since both variables have not received enough exploration.

A meta-analysis of AR-assisted education offers several advantages (Cao and Hsu, 2022 ). Combining the results of multiple studies increases the sample size and statistical power, enabling more accurate and dependable conclusions in AR-assisted education. By analyzing multiple studies together, meta-analysis can identify patterns and trends that may not be apparent in individual studies, indicating the consistency of results across different studies and enhancing the generalizability of findings. Meta-analysis mitigates the impact of bias in individual studies by examining a larger pool of data and reduces the need for replication studies, thereby saving valuable time and resources. It also helps integrate findings with existing theoretical frameworks, providing a more comprehensive understanding of the topic in AR-assisted education. Overall, meta-analysis provides a more robust evidence base for decision-making in policy and practice in AR-assisted education.

The purpose of this meta-analysis is to investigate the impact of Augmented Reality (AR) on educational outcomes while minimizing the aforementioned limitations. We intend to achieve this by incorporating a larger sample size from diverse databases. Our study aims to address the issue of sample bias by expanding the sample size and examining the role of AR features in education. We will include all available studies related to AR, and in cases where adequate information is unavailable, we will reach out to the authors for clarification. Our analysis will also encompass various pedagogical approaches facilitated by AR technologies, with the goal of arriving at comprehensive conclusions regarding attitudes, learning achievements, and motivation.

Literature review

Attitudes toward ar used for education.

The utilization of augmented reality (AR) has been suggested as a means to enhance attitudes towards and satisfaction with education. As reported by Weng et al. ( 2020 ), AR has the potential to induce positive attitudes toward education. Alqarni ( 2021 ) suggests that AR may facilitate positive learning experiences, including academic achievements for students with disabilities. The integration of AR into problem-based learning has also been noted as a promising approach to improving students’ attitudes toward specific subjects (Fidana and Tuncel, 2019 ). Recent research conducted by Sahin and Yilmaz ( 2020 ) found that students who utilized an AR-enhanced science course, specifically “Solar System and Beyond,” exhibited more favorable attitudes toward learning than their non-AR-using peers. Additionally, they reported higher levels of satisfaction and lower levels of anxiety. Delello ( 2014 ) also posits that AR technologies may play a crucial role in improving attitudes toward AR-assisted education.

Despite the potential benefits of AR technology in enhancing attitudes toward education, it is important to acknowledge that some studies have reported negative attitudes toward its use. For instance, Basoglu et al. ( 2018 ) suggest that the use of AR smart glasses (ARSGs) may pose privacy concerns and reduce the perceived ease of use, which can lead to negative attitudes toward AR. Similarly, Akçayır et al. ( 2016 ) assert that students’ lack of familiarity with AR technology can result in frustration and generate negative attitudes toward AR-assisted education. Given the contradictory findings surrounding the impact of AR on attitudes toward education, we propose an alternative hypothesis for further investigation.

H1: The attitudes of learners towards AR-assisted education are significantly more positive compared to those without the aid of AR technologies.

Learning achievements

The majority of studies have reported positive learning outcomes associated with the use of AR technologies. Akçayır and Akçayır ( 2017 ) suggested that utilizing AR technology could enhance learning achievements, foster student engagement, and boost confidence in academic activities. Fidana and Tuncel ( 2019 ) found that integrating AR technologies into problem-based learning approaches resulted in improved learning achievements. Similarly, Sahin and Yilmaz ( 2020 ) reported that students who used AR technologies achieved significantly higher learning outcomes than those who did not. Lee and Hsu ( 2021 ) also demonstrated the efficacy of AR-assisted learning through the “Makeup AR” approach, which enhanced learning achievements, self-efficacy, and reduced cognitive loads. Wu et al. ( 2018 ) further supported the effectiveness of AR-based learning systems, reporting significantly better learning achievements compared to traditional learning methods.

Several studies have reported negative learning outcomes associated with augmented reality (AR) technologies. For instance, Kuhn and Lukowicz ( 2016 ) found that incorporating AR technologies, such as Google Glass, into intelligent classes did not result in significantly higher learning achievements compared to those without AR technologies. Conversely, students who learned using a serious game with AR technologies called Lost in Space demonstrated significantly greater improvements in learning achievements than those who used traditional learning tools, but no significant differences were observed during gameplay (Hou et al., 2021 ). Additionally, AR technologies could potentially have adverse effects on mobile learning achievements, as improper mobile design with AR technologies may lead to frustrating learning outcomes and reduced learning efficiency (Chu, 2014 ; Hwang et al., 2016 ). Given these contradictory results, we propose an alternative hypothesis.

H2. Learning achievements in AR-assisted education exhibit significantly higher results compared to those achieved through non-AR-assisted education.

Motivation of AR technology-assisted learning

Numerous studies have demonstrated that augmented reality (AR) technologies can enhance learning motivation. For example, Cavallo and Laubach ( 2001 ) found that AR technologies could improve learning motivation. Akçayır and Akçayır ( 2017 ) reported that AR technologies motivated students to participate in learning activities. Yildirim ( 2016 ) discovered that students who used computer-based AR technologies were significantly more motivated than the control group who did not use AR technologies. Moreover, Tian et al. ( 2014 ) and Zhang et al. ( 2014 ) indicated that the use of AR technologies in education effectively enhanced students’ motivation. Cen et al. ( 2020 ) observed that a mobile AR-based learning system significantly improved the motivation of secondary chemistry learners. Demitriadou et al. ( 2020 ) suggested that AR technologies were effective in increasing learning motivation.

Despite the positive effects of augmented reality (AR) technologies on learning motivation, some previous studies have shown differing results. For instance, Gómez-García et al. ( 2021 ) found that students who used AR technologies did not exhibit significantly higher learning motivation than those who did not use them. Additionally, Lee and Hsu ( 2021 ) reported that the application of AR in vocational certification courses failed to significantly enhance learning motivation. Furthermore, teachers who resist changing their traditional pedagogical approaches may feel less motivated by AR technologies, which could also dampen students’ motivation for using AR technologies in learning. Similarly, students who are accustomed to traditional learning styles may also exhibit resistance toward AR-assisted learning. Given these implications and inconsistent findings, we propose an alternative hypothesis.

H3. Learning motivation in AR-assisted education shows a substantial increase compared to non-AR-assisted education.

Research methods

This meta-analysis adhered strictly to the protocols outlined by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, as detailed by Page et al. ( 2021 ). PRISMA outlined 27 items that served as a guide throughout the meta-analysis process and provides specific recommendations for conducting a thorough and valid meta-analysis. The ethical committee overseeing this study has granted a waiver for registration, as the study does not involve any human participants and does not violate any ethical criteria.

Eligibility criteria

Following the PRISMA protocol, we established explicit inclusion and exclusion criteria for selecting relevant studies. Inclusion criteria were as follows: (1) large randomized controlled trials that involved AR technology-assisted education and conducted comparative studies; (2) written in English language; and (3) formally and openly published, and peer-reviewed. We excluded studies that (1) focused solely on AR technology without any reference to education; (2) lacked sufficient information for meta-analyses; (3) belonged to the category of review studies; (4) had no relevance to the study topic; (5) were of overall lower quality based on Standards for Reporting on Empirical Social Science Research in AERA Publications; (6) contained insufficient data; (7) had small sample sizes; or (8) yielded unconvincing results.

Search strategy and selection process

The study involved conducting a systematic search of online databases, including Web of Science, Scopus, Wiley, Taylor & Francis, ScienceDirect Elsevier, and SpringerNature, using specific syntactic rules to enter keywords such as “AR, augmented reality, education, control group, experimental group, learning, and teaching”. Prior to the screening, duplicates, records deemed ineligible by automation tools, and those with missing information, small sample sizes, lower quality, lack of sufficient data, or unconvincing conclusions were removed. The selection process was reviewed independently by two researchers, achieving satisfactory inter-rater consistency ( k  = 0.87). In cases of disagreement, a third reviewer was consulted. Ultimately, 28 relevant results were included after screening and excluding ineligible literature (see Fig. 1 ).

A flowchart of the literature inclusion procedure.

Characteristics of the included studies

The present review encompasses studies that showcase the recent accomplishments in AR-assisted education, with publications ranging from 2016 to 2023. The cumulative number of participants in the control group is 1509, while the experimental group consists of 1417 individuals. These studies investigate the comparative effectiveness of AR-assisted and traditional educational approaches in terms of learning achievements, learners’ attitudes, and motivation. All included research articles are published in distinguished journals such as Advances in Physiology Education, Australasian Journal of Educational Technology, Behaviour & Information Technology, British Journal of Educational Technology, Computer Application Engineering Education, Computers & Education, Computers in Human Behavior, Education Sciences, IEEE Transactions on Learning Technologies, Innovation in Language Learning and Teaching, Interactive Learning Environments, International Journal of Human–Computer Interaction, Journal of Baltic Science Education, Journal of Computer Assisted Learning, Journal of Science Education and Technology, and Universal Access in the Information Society (refer to Table 1 ).

Data synthesis

In order to ensure the reliability of our findings, we employed two methods: publication bias testing and sensitivity analyses. Publication bias is a common issue in research, as journals tend to prioritize publishing positive results over negative ones. To detect potential publication bias, we utilized Begg’s (Begg and Mazumdar, 1994 ) and Egger’s tests (Egger et al., 1997 ). We also examined the distribution of individual studies to identify any presence or absence of publication bias. Additionally, we performed sensitivity analyses using Stata/MP 14.0 software to further validate our results.

Begg’s and Egger’s tests are two commonly used statistical methods to assess publication bias in meta-analyses. Begg’s test is a rank correlation test that examines the association between effect sizes and their variances or standard errors. A non-significant p -value (e.g., p  > 0.05) suggests that there is no evidence of publication bias. However, a significant p -value (e.g., p  < 0.05) may indicate the presence of publication bias, but it can also mean that the sample size is too small or the number of studies included in the analysis is too few. Egger’s test is a linear regression test that examines the association between the effect sizes and their precision (the reciprocal of variance). A non-significant p -value (e.g., p  > 0.05) indicates that there is no evidence of publication bias. However, a significant p -value (e.g., p  < 0.05) suggests the presence of publication bias, but it can also mean that the sample size is too small, or there is substantial heterogeneity among the included studies.

The present meta-analysis was conducted using Stata/MP 14.0 software. Firstly, we extracted data pertaining to mean values, standard deviations, and participant numbers across both experimental and control groups. Additionally, subgroups such as learning achievements, attitudes, and motivation in AR-assisted education were also extracted. Effect sizes were then calculated using Cohen’s d formula: d  = Me−Mc/Sp, where Me represents the means of the experimental group, Mc represents the means of the control group, and Sp signifies the pooled standard deviation of both groups (Sedgwick and Marston, 2013 ). We will classify effect size values as very small if they are around 0.1, small if approximately 0.2, medium if roughly 0.5, large if about 0.8, very large if near 1.2, and huge if approaching 2 (Sawilowsky, 2009 ).

The heterogeneity of estimates was assessed by the researchers using I 2 , Q , z , and p values. The degree of heterogeneity was categorized as unimportant if I 2 was <40%, moderate if I 2 was between 30% and 60%, substantial if I 2 was between 50% and 90%, and considerable if it ranged from 75% to 100% (Higgins and Green, 2021 ). We employed a random-effect model for meta-analysis if I 2 was >50%, and a fixed-effect model if I 2 was <50%. In addition to I 2 , Q , z , and p values were also considered in determining the level of heterogeneity.

In cases where a single study produced multiple results, we utilized the Statistics Toolkit (STATTOOLS) to merge participant numbers, means, and standard deviations into a single group (Altman et al., 2000 ). We combined various subgroups such as attitudes (Alqarni, 2021 ; Fidana and Tuncel, 2019 ; Sahin and Yilmaz, 2020 ), attractiveness (Albrecht et al., 2013 ), learning interest (Chin and Wang, 2021 ), satisfaction (Huang et al., 2021 ; Ucar et al., 2017 ; Wu et al., 2018 ), and self-efficacy (Lee and Hsu, 2021 ) under the “attitudes” category. The “learning achievements” subgroup included test scores (e.g. Gonzalez et al., 2020 ), academic achievement, academic averages (Selek and Kiymaz, 2020 ), evaluation scores (Gargrish et al., 2021 ), final exam scores (Gonzalez et al., 2020 ), grades of work, financial knowledge (Candra Sari et al., 2021 ), learning outcomes (Stojanović et al., 2020 ), learning performance (Hanafi et al., 2016 ), the mathematical calculation (Ruiz-Ariza et al., 2018 ), operational effectiveness (Mao and Chen, 2021 ), spatial perception skills (Carbonell Carrera and Bermejo Asensio, 2017 ), test and quiz scores (Christopoulos et al., 2021 ), visualization skills (Omar et al., 2019 ), and writing skills (Wang, 2017a ). The “motivation” subgroup focused on learning motivation (Chang et al., 2016 ; Chu et al., 2019 ; Gómez-García et al., 2021 ; Lee and Hsu, 2021 ; Christopoulos et al., 2021 ). The included studies utilized AR technologies in education as the treatment.

If multiple experimental groups were used, preference would be given to the group that was most closely associated with the use of augmented reality (AR). Among the experimental groups that utilized AR, priority would be given to the group that had the most stringent design and provided the most compelling results. When selecting a control group, the one that could provide the most informative comparative results with the experimental group would be selected. In studies where pre- and post-tests were conducted to compare control and experimental groups, data from the post-tests that underwent the treatment would be retrieved.

The sample size, methodological quality, and age of participants can all contribute to the variability of effects observed in a meta-analysis. Larger sample sizes generally lead to more precise estimates of effect size with less variance. Small samples may have greater variability due to sampling error. Studies that are well-designed and implemented with appropriate controls tend to produce more reliable and valid results. Poorly designed studies with bias or confounding factors can produce less trustworthy outcomes and introduce heterogeneity in the meta-analysis. Studies that include participants from different age groups may lead to variations in treatment effects. For example, an intervention may work better for younger individuals but not as well for older populations. Therefore, in this meta-analysis, differences in sample size, methodological quality, and age of participants across studies may have negatively influenced the generalizability of the results.

Testing for hypotheses

H1. The attitudes of learners towards AR-assisted education are significantly more positive compared to those without the aid of AR technologies .

In a random-effect model, the variance is assumed to consist of two components: within-group variation and between-group variation. The group-specific effects are considered random variables that follow a normal distribution with a mean zero and a certain variance. In contrast, a fixed-effect model assumes that each group has its own fixed effect, which is not normally distributed. The interpretation of results from a random-effect model is usually more generalizable than from a fixed-effect model since it accounts for both within-group and between-group variation. However, a random-effect model may have less statistical power than a fixed-effect model when there are only a few groups or when the within-group variability is small. Therefore, the choice between the two models depends on the research question and the specific data characteristics.

The effect model used for conducting the meta-analysis was determined based on the level of heterogeneity. The observed variances in study outcomes across studies were attributed to heterogeneity rather than random errors, specifically in relation to attitudes towards AR-assisted education (indicated by Q  = 171.78, I 2  = 94.2, p  < 0.01 in Table 2 ). As a result, random-effect models were employed to analyze attitudes within the context of AR-assisted education using meta-analytic techniques.

A forest plot was generated using Stata/MP 14.0 software to test the alternative hypotheses (Fig. 2 ). The plot included 11 effect sizes, with individual studies represented by dots in the middle column and the horizontal line indicating 95% confidence intervals. The no-effect line was represented by the middle line, while the diamond at the bottom indicated the pooled result. If the horizontal line or diamond crossed the no-effect line, it suggested non-significant differences. The diamond was located to the right of the middle line, indicating a significantly more favorable attitude in the experimental group compared to the control ( d  = 1.08, 95% CI = 0.44–1.72, z  = 3.32, p  = 0.001 in Table 2 ).

A forest plot of differences in attitudes between control and experimental groups.

To test for publication bias, a funnel plot was created using the same software. Figure 3 shows symmetrically distributed dots along both sides of the middle line, suggesting the absence of publication bias ( z  = 1.63, p  = 0.102 through Begg’s test in Table 3 ). Therefore, researchers accept the first alternative hypotheses.

A funnel plot of publication bias in attitudes.

H2. Learning achievements in AR-assisted education exhibit significantly higher results compared to those achieved through non-AR-assisted education .

In terms of learning achievements, the estimations yielded significant heterogeneity ( Q  = 281.66, p  < 0.01, I 2  = 92.5 in Table 2 ), prompting the researchers to employ a random-effect model for the meta-analysis. The results indicated a significant difference between the experimental and control groups, with the former achieving significantly higher learning outcomes ( d  = 0.85, 95% CI = 0.47–1.22, z  = 4.37, p  < 0.01 in Table 2 and Fig. 4 ). Additionally, there was no indication of publication bias in the data according to the funnel plot analysis (Fig. 5 ) and Begg’s test ( z  = 1.75, p  = 0.08 in Table 3 ), thus leading the researchers to accept the second alternative hypothesis.

A forest plot of differences in learning achievements between control and experimental groups.

A funnel plot of publication bias in learning achievements.

H3. Learning motivation in AR-assisted education shows a substantial increase compared to non-AR-assisted education .

In order to test the alternate hypothesis, researchers utilized a random-effects model for conducting meta-analysis due to significant heterogeneity in estimates ( Q  = 12.52, p  = 0.028, I 2  = 60.1). A forest plot (Fig. 6 ) was created which showed that the pooled estimate of motivation, represented by the diamond, intersected with the no-effect line, indicating no significant difference in motivation between the two groups ( d  = 0.85, 95% CI = 0.47–1.22, z  = 4.37, p  < 0.01 in Table 2 and Fig. 6 ). Additionally, results from Begg’s test ( z  = 1.13, p  = 0.26) and Egger’s test ( z  = 1.18, p  = 0.302 in Table 3 ) depicted symmetric distribution of dots on either side of the middle line in Fig. 7 , thereby indicating no presence of publication bias. Consequently, the third alternative hypothesis was rejected by the researchers.

A forest plot of differences in motivation between control and experimental groups.

A funnel plot of publication bias in motivation.

In order to verify the reliability of our estimate results, we performed sensitivity analyses using the Stata/MP 14.0 program by entering the command “metaninf N M SD N0 M0 SD0, random cohen”. The results are presented in Fig. 8 , where each dot represents an individual study, while the middle line displays the effect size and the lines on both sides represent the upper and lower confidence interval limits. All of the dots fall within the given confidence interval limits when a particular study is excluded. We conducted separate sensitivity analyses for attitudes, learning achievements, and motivation, and obtained the same results, indicating that the overall and separate estimates of our study are reliable and robust. The final results are summarized in Table 4 .

Results of the sensitivity analysis.

Attitudes toward AR for educational purposes

It can be concluded that students exhibit more favorable attitudes towards AR-assisted education than traditional education. Implementing AR technologies in education has the potential to generate excitement and interest among learners, leading to positive attitudes toward AR-assisted learning. This is especially true for those who experience AR technologies for the first time, as they may find the technology curious and even magical (Sahin and Yilmaz, 2020 ; Akram et al., 2021 ). AR technologies have three dimensions that provide students with a more tangible and authentic learning experience, ultimately enhancing learning effectiveness (Wojciechowski and Cellary, 2013 ). AR technologies capture students’ attention, increase their engagement, and immerse them in educational activities, leading to positive attitudes toward AR-assisted education (Perez-Lopez and Contero, 2013 ). Positive attitudes towards AR-assisted education are closely linked to learning achievements in AR contexts (Sahin and Yilmaz, 2020 ). This positive correlation may further reinforce positive attitudes as students’ learning achievements significantly improve when compared to those achieved through traditional learning.

It is reasonable to expect that AR-assisted education can result in significantly higher learning achievements compared to traditional education. The multi-dimensional scaffolding functions of AR technologies may offer novel experiences and stimulate students to participate in the learning process, thereby enhancing their learning achievements (Gilliam et al., 2017 ). AR-assisted learning may also foster students’ curiosity, which can increase their cognitive effort and improve their learning achievements (Kuhn and Lukowicz, 2016 ). Strong curiosity may help students focus on learning content and reduce distractions, leading to improved learning outcomes. In AR-assisted contexts, students typically experience lower cognitive loads than those without the use of AR technologies and also report higher levels of satisfaction (Wu et al., 2018 ). This may further contribute to improved learning achievements facilitated by AR technologies.

Although this study did not find a significant difference in motivation levels between AR-assisted education and traditional methods, it is reasonable to expect such a difference based on the potential benefits of AR technologies. The remarkable functions of AR technologies may encourage students to engage in simulated learning activities and associate virtual with real learning environments (Abdullah, 2022 ), leading to increased learning motivation and the development of positive attitudes towards learning (Tian et al., 2014 ). Students tend to enjoy using AR technologies in their learning, finding them easy and convenient to use, and they report high satisfaction with their AR-assisted learning experiences (Ozarslan, 2013 ), which can reduce their learning anxiety compared to traditional learning (Tomi and Rambli, 2013 ; Al-Ansi, 2021 ). Thus, students are motivated to continue using AR technologies to enhance their learning experiences. Lee and Hsu’s ( 2021 ) failure to detect significant differences in motivation levels might be due to the short duration of their experiment, poor Internet connection, or the use of small smartphones that could hinder students’ ability to effectively utilize AR technologies.

Major findings

The results of this study are in line with previous research (e.g. Christopoulos et al., 2021 ; Carbonell Carrera and Bermejo Asensio, 2017 ), indicating that AR-assisted education generates more positive attitudes among learners and leads to higher learning achievements compared to traditional methods. However, the study did not observe any significant differences in motivation levels between AR-assisted education and non-AR-assisted education. The study authors explored several explanations for this unexpected finding.

Limitations

This study has several limitations. Firstly, due to constraints in the availability of library resources, it was not possible to access all relevant literature. Secondly, Begg’s and Egger’s tests indicate that publication bias exists regarding learning achievements in AR-assisted education, which may reduce the reliability of the findings. Additionally, the variability of research contexts makes it challenging to fully summarize the effects of AR technologies on educational outcomes.

Future research directions

Other factors, such as learning styles and learner personality, may also significantly impact the effects of AR technologies on educational outcomes. Future research could incorporate a more comprehensive range of influencing factors. Additionally, future studies could explore the differences between the application of mobile and static AR technologies in educational contexts (Lee and Hsu, 2021 ). Researchers should also consider the impact of technostress, interaction, affection, cognition, and telepresence on AR-assisted learning experiences and achievements (Baabdullah et al., 2022 ). Furthermore, studies could focus on the effects of AR on learners’ spatial ability (Di and Zheng, 2022 ).

Data availability

The datasets generated during and/or analyzed during the current study are openly at https://osf.io/jfwb2/?view_only=872843fa65cf4d35b35afb7214b793b9 .

Abdullah MAA (2022) Investigating characteristics of learning environments during the COVID-19 pandemic: a systematic review. Can J Learn Technol 48(1):1–27

Google Scholar  

Akçayır M, Akçayır G (2017) Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educ Res Rev 20:1–11. https://doi.org/10.1016/j.edurev.2016.11.002

Article   Google Scholar  

Akçayır M, Akçayır G, Pektaş HM, Ocak MA (2016) Augmented reality in science laboratories: the effects of augmented reality on university students’ laboratory skills and attitudes toward science laboratories. Comput Hum Behav 57:334–342. https://doi.org/10.1016/j.chb.2015.12.054

Akram H, Yingxiu Y, Al-Adwan AS, Alkhalifah A (2021) Technology integration in higher education during COVID-19: an assessment of online teaching competencies through technological pedagogical content knowledge model. Front Psychol 12:736522. https://doi.org/10.3389/fpsyg.2021.736522

Article   PubMed   PubMed Central   Google Scholar  

Al-Ansi AM (2021) Students anxiety and recruitment during Covid-19 pandemic: role of university, specialization and employment expectation. Perspect Sci Educ 49(1):403–413. https://doi.org/10.32744/pse.2021.1.27

Albrecht UV, Folta-Schoofs K, Behrends M, Von Jan U (2013) Effects of mobile augmented reality learning compared to textbook learning on medical students: randomized controlled pilot study. J Med Internet Res 15(8):e182. https://doi.org/10.2196/jmir.2497

Alqarni T (2021) Comparison of augmented reality and conventional teaching on special needs students’ attitudes towards science and their learning outcomes. J Balt Sci Educ 20(4):558–572. https://doi.org/10.33225/jbse/21.20.558

Altman DG, Machin D, Bryant TN, Gardner MJ (2000) Statistics with confidence, 2nd edn. BMJ Books, pp. 28–31

Baabdullah AM, Alsulaimani AA, Allamnakhrah A, Alalwan AA, Dwivedi YK, Rana NP (2022) Usage of augmented reality (AR) and development of e-learning outcomes: an empirical evaluation of students’ e-learning experience. Comput Educ 177. https://doi.org/10.1016/j.compedu.2021.104383

Basoglu N, Goken M, Dabic M, Ozdemir Gungor D, Daim TU (2018) Exploring adoption of augmented reality smart glasses: applications in the medical industry. Front Eng Manag 5(2):167–181. https://doi.org/10.15302/j-fem-2018056

Begg CB, Mazumdar M (1994) Operating characteristics of a rank correlation test for publication bias. Biometrics 50(4):1088–1101

Article   CAS   PubMed   MATH   Google Scholar  

Bölek KA, De Jong G, Henssen D (2021) The effectiveness of the use of augmented reality in anatomy education: a systematic review and meta-analysis. Sci Rep 11(1). https://doi.org/10.1038/s41598-021-94721-4

Bork F, Lehner A, Eck U, Navab N, Waschke J, Kugelmann D (2020) The effectiveness of collaborative augmented reality in gross anatomy teaching: a quantitative and qualitative pilot study. Anat Sci Educ. https://doi.org/10.1002/ase.2016

Bursali H, Yilmaz RM (2019) Effect of augmented reality applications on secondary school students’ reading comprehension and learning permanency. Comput Hum Behav 95:126–135. https://doi.org/10.1016/j.chb.2019.01.035

Cai S, Liu C, Wang T, Liu E, Liang J (2021) Effects of learning physics using Augmented Reality on students’ self-efficacy and conceptions of learning. Br J Educ Technol 52(1):235–251. https://doi.org/10.1111/bjet.13020

Candra Sari R, Rika Fatimah PL, Ilyana S, Dwi Hermawan H (2021) Augmented reality (AR)-based sharia financial literacy system (AR-SFLS): a new approach to virtual sharia financial socialization for young learners. Int J Islam Middle East Finance Manag. https://doi.org/10.1108/IMEFM-11-2019-0484

Cao X, Hsu Y (2022) Systematic review and meta-analysis of the impact of virtual experiments on students’ learning effectiveness. Interact Learn Environ. https://doi.org/10.1080/10494820.2022.2072898

Carbonell Carrera C, Bermejo Asensio LA (2017) Landscape interpretation with augmented reality and maps to improve spatial orientation skill. J Geogr High Educ 41(1):119–133. https://doi.org/10.1080/03098265.2016.1260530

Cavallo AM, Laubach TA (2001) Students’ science perceptions and enrollment decisions in differing learning cycle classrooms. J Res Sci Teach 38(9):1029–1062. https://doi.org/10.1002/tea.1046

Cen L, Ruta D, Mohd LM, Ng J (2020) Augmented Immersive Reality (AIR) for improved learning performance: a quantitative evaluation. IEEE Trans Learn Technol 13:283–296. https://doi.org/10.1109/TLT.2019.2937525

Chang RC, Chung LY, Huang YM (2016) Developing an interactive augmented reality system as a complement to plant education and comparing its effectiveness with video learning. Interact Learn Environ 24(6):1245–1264. https://doi.org/10.1080/10494820.2014.982131

Chin KY, Wang CS (2021) Effects of augmented reality technology in a mobile touring system on university students’ learning performance and interest. Australas J Educ Technol 37(1):27–42. https://doi.org/10.14742/ajet.5841

Christopoulos A, Pellas N, Kurczaba J, Macredie R (2021) The effects of augmented reality‐supported instruction in tertiary‐level medical education. Br J Educ Technol. https://doi.org/10.1111/bjet.13167

Chu HC (2014) Potential negative effects of mobile learning on students’ learning achievement and cognitive load—a format assessment perspective. J Educ Technol Soc 17(1):332–344. http://www.jstor.org/stable/jeductechsoci.17.1.332

Chu HC, Chen JM, Hwang GJ, Chen TW (2019) Effects of formative assessment in an augmented reality approach to conducting ubiquitous learning activities for architecture courses. Univers Access Inf Soc 18(2):221–230. https://doi.org/10.1007/s10209-017-0588-y

Ciloglu T, Ustun AB (2023) The effects of mobile AR-based biology learning experience on students’ motivation, self‐efficacy, and attitudes in online learning. J Sci Educ Technol. https://doi.org/10.1007/s10956-023-10030-7

Delello JA (2014) Insights from pre-service teachers using science-based augmented reality. J Comput Educ 1(4):295–311. https://doi.org/10.1007/s40692-014-0021-y

Demitriadou E, Stavroulia KE, Lanitis A (2020) Comparative evaluation of virtual and augmented reality for teaching mathematics in primary education. Educ Inf Technol 25:381–401. https://doi.org/10.1007/s10639-019-09973-5

Di X, Zheng X (2022) A meta-analysis of the impact of virtual technologies on students’ spatial ability. Educ Technol Res Dev. https://doi.org/10.1007/s11423-022-10082-3

Egger M, Smith GD, Schneider M, Minder C (1997) Bias in meta-analysis detected by a simple, graphical test. Br Med J 315:629–634. https://www.jstor.org/stable/25175671

Article   CAS   Google Scholar  

Fidana M, Tuncel M (2019) Integrating augmented reality into problem based learning: the effects on learning achievement and attitude in physics education. Comput Educ 142:103635. https://doi.org/10.1016/j.compedu.2019.103635

Gargrish S, Kaur DP, Mantri A, Singh G, Sharma B (2021) Measuring effectiveness of augmented reality-based geometry learning assistant on memory retention abilities of the students in 3D geometry. Comput Appl Eng Educ. https://doi.org/10.1002/cae.22424

Garzón J, Acevedo J (2019) Meta-analysis of the impact of Augmented Reality on students’ learning gains. Educ Res Rev 27:244–260. https://doi.org/10.1016/j.edurev.2019.04.001

Garzón J, Kinshuk, Baldiris S, Gutiérrez J, Pavón J (2020) How do pedagogical approaches affect the impact of augmented reality on education? A meta-analysis and research synthesis. Educ Res Rev 31:100334. https://doi.org/10.1016/j.edurev.2020.100334

Garzón J, Pavón J, Baldiris S (2019) Systematic review and meta-analysis of augmented reality in educational settings. Virtual Real 23(4):447–459. https://doi.org/10.1007/s10055-019-00379-9

Gilliam M, Jagoda P, Fabiyi C, Lyman P, Wilson C, Hill B, Bouris A (2017) Alternate reality games as an informal learning tool for generating STEM engagement among underrepresented youth: a qualitative evaluation of the source. J Sci Educ Technol 26(3):295–308. https://doi.org/10.1007/s10956-016-9679-4

Gómez-García G, Hinojo-Lucena FJ, Alonso-García S, Romero-Rodríguez JM (2021) Mobile learning in pre-service teacher education: perceived usefulness of AR technology in primary education. Educ Sci 11(6):275. https://doi.org/10.3390/educsci11060275

Gonzalez AA, Lizana PA, Pino S, Miller BG, Merino C (2020) Augmented reality-based learning for the comprehension of cardiac physiology in undergraduate biomedical students. Adv Physiol Educ 44(3):314–322. https://doi.org/10.1152/advan.00137.2019

Article   PubMed   Google Scholar  

Hanafi HFB, Said CS, Ariffin AH, Zainuddin NA, Samsuddin K (2016) Using a collaborative Mobile Augmented Reality learning application (CoMARLA) to improve Improve Student Learning. IOP Conf Ser 160:012111. https://doi.org/10.1088/1757-899x/160/1/012111

Higgins JPT, Green S (2021) Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration 2011. https://handbook-5-1.cochrane.org/

Hou HT, Fang YS, Tang JT (2021) Designing an alternate reality board game with augmented reality and multi-dimensional scaffolding for promoting spatial and logical ability. Interact Learn Environ 1–21. https://doi.org/10.1080/10494820.2021.1961810

Huang HM, Huang TC, Cheng CY (2021) Reality matters? exploring a tangible user interface for augmented-reality-based fire education. Univers Access Inf Soc. https://doi.org/10.1007/s10209-021-00808-0

Hwang GJ, Wu PH, Chen CC, Tu NT (2016) Effects of an augmented reality-based educational game on students’ learning achievements and attitudes in real-world observations. Interact Learn Environ 24(8):1895–1906. https://doi.org/10.1080/10494820.2015.1057747

Kuhn J, Lukowicz P (2016) gPhysics—using smart glasses for head-centered, context-aware learning in physics experiments. IEEE Trans Learn Technol 9(4):304–317. https://doi.org/10.1109/TLT.2016.2554115

Lee CJ, Hsu Y (2021) Sustainable education using augmented reality in vocational certification courses. Sustainability 13(11):6434. https://doi.org/10.3390/su13116434

Liu YC, Huang TH, Lin IH (2022) Hands-on operation with a Rolling Alphabet-AR System improves English learning achievement. Innov Language Learn Teach. https://doi.org/10.1080/17501229.2022.2153852

Mao CC, Chen CH (2021) Augmented reality of 3D content application in common operational picture training system for army. Int J Hum–Comput Interact 1–17. https://doi.org/10.1080/10447318.2021.1917865

Najmi AH, Alhalafawy WS, Zaki MZT (2023) Developing a sustainable environment based on augmented reality to educate adolescents about the dangers of electronic gaming addiction. Sustainability 15(4):3185. https://doi.org/10.3390/su15043185

Omar M, Ali DF, Mokhtar M, Zaid NM, Jambari H, Ibrahim NH (2019) Effects of mobile augmented reality (MAR) towards students’ visualization skills when learning orthographic projection. Int J Emerg Technol Learn 14(20):106. https://doi.org/10.3991/ijet.v14i20.11463

Ozarslan Y (2013) Effects of the effect of learning materials that are enriched through extended reality on student’s achievement and satisfaction. Doctoral Thesis. Anadolu University Social Sciences Institute, Eskis¸ehir

Ozdemir M, Sahin C, Arcagok S, Demir MK (2018) The effect of augmented reality applications in the learning process: a meta-analysis study. Eurasian J Educ Res 74:165–186. https://doi.org/10.14689/ejer.2018.74.9

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD et al. (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. J Clin Epidemiol 75(2):192–192. https://doi.org/10.1016/j.rec.2021.10.019

Perez-Lopez D, Contero M (2013) Delivering educational multimedia contents through an augmented reality application: a case study on its impact on knowledge acquisition and retention. Turk Online J Educ Technol 12(4):19–28. https://doi.org/10.2307/3426023

Ruiz-Ariza A, Casuso RA, Suarez-Manzano S, Martínez-López EJ (2018) Effect of augmented reality game Pokémon GO on cognitive performance and emotional intelligence in adolescent young. Comput Educ 116:49–63. https://doi.org/10.1016/j.compedu.2017.09.002

Sahin D, Yilmaz RM (2020) The effect of Augmented Reality Technology on middle school students’ achievements and attitudes towards science education. Comput Educ 144:103710. https://doi.org/10.1016/j.compedu.2019.103710

Saleem M, Kamarudin S, Shoaib HM, Nasar A (2021) Influence of augmented reality app on intention towards e-learning amidst COVID-19 pandemic. Interact Learn Environ. https://doi.org/10.1080/10494820.2021.1919147

Sawilowsky SS (2009) New effect size rules of thumb. J Modern Appl Stat Methods 8(2):597–599. https://doi.org/10.22237/jmasm/1257035100

Article   MathSciNet   Google Scholar  

Sedgwick P, Marston L (2013) Meta-analyses: standardised mean differences. Br Med J 347:f7257. https://doi.org/10.1136/bmj.f7257

Selek M, Kiymaz YE (2020) Implementation of the augmented reality to electronic practice. Comput Appl Eng Educ 28(2):420–434. https://doi.org/10.1002/cae.22204

Stojanović D, Bogdanović Z, Petrović L, Mitrović S, Labus A (2020) Empowering learning process in secondary education using pervasive technologies. Interact Learn Environ 1–14. https://doi.org/10.1080/10494820.2020.1806886

Tezer M, Yıldız EP, Masalimova ARR, Fatkhutdinova AM, Zheltukhina MRR, Khairullina ER(2019)Trends of augmented reality applications and research throughout the world: meta-analysis of theses, articles and papers between 2001–2019 years Int J Emerg Technol Learn 14(22):154. https://doi.org/10.3991/ijet.v14i22.11768

Tian K, Endo M, Urata M, Mouri K, Yasuda T (2014) Multi-viewpoint smartphone AR-based learning system for astronomical observation. Int J Comput Theory Eng 6(5):396–400. https://doi.org/10.3991/ijim.v8i3.3731

Tomi AB, Rambli DRA (2013) An interactive mobile augmented reality magical playbook: learning number with the thirsty crow. Procedia Comput Sci 25:123–130. https://doi.org/10.1016/j.procs.2013.11.015

Ucar E, Ustunel H, Civelek T, Umut I (2017) Effects of using a force feedback haptic augmented simulation on the attitudes of the gifted students towards studying chemical bonds in virtual reality environment. Behav Inf Technol 36(5):540–547. https://doi.org/10.1080/0144929x.2016.1264483

Wang YH (2017a) Exploring the effectiveness of integrating augmented reality-based materials to support writing activities. Comput Educ 113:162–176. https://doi.org/10.1016/j.compedu.2017.04.013

Wang YH (2017b) Using augmented reality to support a software editing course for college students. J Comput Assist Learn 33:532–546. https://doi.org/10.1111/jcal.12199

Weng C, Otanga S, Christianto S, Chu R (2020) Enhancing student’s biology learning by using augmented reality as a learning supplement. J Educ Comput Res 58(4):747–770. https://doi.org/10.1177/0735633119884213

Wojciechowski R, Cellary W (2013) Evaluation of learners’ attitude toward learning in ARIES augmented reality environments. Comput Educ 68:570–585. https://doi.org/10.1016/j.compedu.2013.02.014

Wu PH, Hwang GJ, Yang ML, Chen CH (2018) Impacts of integrating the repertory grid into an augmented reality-based learning design on students’ learning achievements, cognitive load and degree of satisfaction. Interact Learn Environ 26(2):221–234. https://doi.org/10.1080/10494820.2017.1294608

Yang FCO, Lai HM, Wang YW (2022) Effect of augmented reality-based virtual educational robotics on programming students’ enjoyment of learning, computational thinking skills, and academic achievement. Comput Educ 195:104721. https://doi.org/10.1016/j.compedu.2022.104721

Yildirim S (2016) The effect of augmented reality applications in Science courses on students’ achievement, motivation, perception towards problem solving skills and attitudes. Non-published master’s thesis, Ankara University Institute of Educational Sciences, Ankara

Yousef AMF (2021) Augmented reality assisted learning achievement, motivation, and creativity for children of low‐grade in primary school. J Comput Assist Learn 37(4):966–977. https://doi.org/10.1111/jcal.12536

Zhang J, Sung YT, Hou HT, Chang KE (2014) The development and evaluation of an augmented reality-based armillary sphere for astronomical observation instruction. Comput Educ 73:178–188. https://doi.org/10.1016/j.compedu.2014.01.003

Download references

Acknowledgements

The authors extend gratitude for funding support from the following: Shan Dong Humanities and Social Sciences Project in 2022 (Grant No: 2022-JCJY-09): A Study on English College Instructors' Leadership in China, funded by Shandong Federation of Social Sciences; 2019 MOOC of Beijing Language and Culture University (MOOC201902) (Important) “Introduction to Linguistics”; “Introduction to Linguistics” of online and offline mixed courses in Beijing Language and Culture University in 2020; Special fund of Beijing Co-construction Project-Research and reform of the “Undergraduate Teaching Reform and Innovation Project” of Beijing higher education in 2020-innovative “multilingual +” excellent talent training system (202010032003); the research project of Graduate Students of Beijing Language and Culture University “Xi Jinping: The Governance of China” (SJTS202108).

Author information

Authors and affiliations.

Department of College English Teaching, Qufu Normal University, No. 80 Yantai Road, Donggang District, 276826, Rizhao City, China

Faculty of International Studies, Beijing Language and Culture University, 15 Xueyuan Road, Haidian District, 100083, Beijing, China

Zhonggen Yu

You can also search for this author in PubMed   Google Scholar

Contributions

WC collected data, analyzed data, edited, revised, and confirmed the manuscript; ZY conceptualized and supervised the research process.

Corresponding author

Correspondence to Zhonggen Yu .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Informed consent

Additional information.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Cao, W., Yu, Z. The impact of augmented reality on student attitudes, motivation, and learning achievements—a meta-analysis (2016–2023). Humanit Soc Sci Commun 10 , 352 (2023). https://doi.org/10.1057/s41599-023-01852-2

Download citation

Received : 02 December 2022

Accepted : 13 June 2023

Published : 26 June 2023

DOI : https://doi.org/10.1057/s41599-023-01852-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Unlocking the cosmos: evaluating the efficacy of augmented reality in secondary education astronomy instruction.

• Enzo Ferrari
• Pablo Herrero Teijón
• Camilo Ruiz

Journal of New Approaches in Educational Research (2024)

Analyzing the Effective Use of Augmented Reality Glasses in University Physics Laboratory Courses for the Example Topic of Optical Polarization

• Daniel Laumann
• Paul Schlummer
• Stefan Heusler

Journal of Science Education and Technology (2024)

The effects of using augmented reality on vocabulary learning and attitude of pre-school children in English education

• Fatma Burcu Topu
• Rabia Meryem Yilmaz
• Ayşegül Takkaç Tulgar

Education and Information Technologies (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Combining augmented reality and gamification to enhance engineering education

Cited by counts, online attention, usage metrics, similar items.

• Investigating virtual reality as a situated learning tool for supporting general education Author(s): Zhao, Zhenjie 2020
• Investigating Visualization Techniques for Observing a Group of Virtual Reality Users using Augmented Reality Author(s): Thanyadit, Santawat; Punpongsanon, Parinya; Pong, Ting-Chuen 2019
• VisionARi: An Online Community-Based Augmented Reality Platform for Immersive Learning Author(s): Park, Jungjin; Fattah, Ryu; Li, Larry K.B. 2022
• Practical mobile edge computing : in the cases of augmented and virtual reality Author(s): Zhang, Wenxiao CSE 2019
• CollectiAR: Computer Vision-Based Word Hunt for Children with Dyslexia Author(s): Fei, Danlu; Gao, Ze; Yuan, Linping... 2022

• Open access
• Published: 04 February 2019

Perspectives on how to evaluate augmented reality technology tools for education: a systematic review

• Manoela M. O. da Silva   ORCID: orcid.org/0000-0001-5401-6540 1 ,
• João Marcelo X. N. Teixeira 1 , 2 ,
• Patrícia S. Cavalcante 3 &
• Veronica Teichrieb 1  

Journal of the Brazilian Computer Society volume  25 , Article number:  3 ( 2019 ) Cite this article

18k Accesses

39 Citations

Metrics details

Education has benefited from augmented reality’s (AR) potential to promote interactive experiences both inside and outside the classroom. A systematic review was conducted on how AR’s impact in the learning process has been evaluated. We selected papers from 2009 to 2017 in three databases, IEEE, ACM, and Science Direct, using an open-source crawler, and in one Brazilian Conference, SBIE. We followed the PRISMA protocol. Forty-five works were selected and used to extract data for our research. They were also analyzed according to quantitative and qualitative criteria. The results from all the papers are available in an online database. Results evidenced an increase in the number of papers evaluating the AR’s impact in education. They also showed that AR has been applied in different areas and contexts. Most papers reported positive outcomes as a result of AR insertion. However, most studies lacked the involvement of the teacher and the use of multiple metrics to evaluate educational gains.

Introduction

Augmented reality (AR) is a technology that consists of adding virtual elements to a real scene coherently so that ideal users cannot differentiate them from the real scene [ 3 ]. Although all fields of knowledge can potentially take advantage from AR, Tori et al. [ 72 ] argue that education will be particularly modified by its introduction. The coexistence of virtual and real environments allows learners to experience phenomena that otherwise would be impossible in the real world. This allows learners to visualize complex spatial relationships and abstract concepts and, therefore, develop important abilities that cannot be evolved in other technology learning environments [ 78 ].

It has been long since AR’s potential in education has been investigated. According to Kostaras et al. [ 41 ], AR can aid learning and make the overall process more interesting and pleasant. In a rapidly changing society as ours where there is a great amount of information available, it is of major importance to know how to locate information and use it efficiently. AR dramatically shifts the location and timing of education and training [ 46 ].

Billinghurst and Duenser [ 5 ] explain that unlike other computer interfaces that draw users away from the real world and onto the screen, AR interfaces enhance the real world experience as shown in Fig.  1 , which presents an AR application designed to create new museum experiences [ 2 ]. Billinghurst and Duenser [ 5 ] also highlight some reasons why AR educational experiences are different: (a) support of seamless interaction between real and virtual environments, (b) use of a tangible interface metaphor for object manipulation, and (c) ability to transition smoothly between reality and virtuality.

AR application developed to enhance museum experience

Although AR has been studied for over 40 years, only in the last decade it began to be formally evaluated [ 23 , 24 , 68 ]. One of the reasons why it took so long to have user evaluations may be a lack of knowledge on how to properly evaluate AR experiences and design experiments [ 24 ]. Dünser et al. [ 24 ] claim that there seems to be a lack of understanding regarding the need of doing studies and the right motivation for carrying them. If user evaluations are conducted out of incorrect motivation or if empirical methods are not properly applied, the findings are of limited value or can even be misleading.

Until that time, the amount of AR systems formally evaluated was rather small [ 23 ]. Swan and Gabbard [ 68 ] and Dünser et al. [ 24 ] have found that only around 8% of published AR research papers included formal evaluations. According to Dünser and Billinghurst [ 22 ], one reason for this small percentage may be the lack of suitable methods for evaluating AR interfaces. Researchers in non-conventional interface fields such as virtual reality (VR) or AR cannot rely solely on design guidelines for traditional user interfaces since new interfaces afford new forms of interactions [ 22 ]. Since then, more works address some form of user evaluation [ 6 ].

When dealing with educational AR systems, it is also important to evaluate the impact of learning applications and the feasibility of incorporating them into the classrooms. Many factors are involved in this process varying from cost to staff’s acceptance. Evaluation of technology is an important step in design instruction, which is the process by which learning products and experiences are designed, developed, and delivered Footnote 1 . Also, it is necessary to evaluate it properly so practitioners are more confident in its positive effects. It is also relevant to consider the points of view of both teachers and learners since they might differ.

In the last decade, a few papers have been published evaluating educational aspects of AR applications used for education. For instance, Balog and Pribeanu [ 4 ] have shown the same aspect can be valued differently by both teachers and learners. One survey reviewed applications intended to complement traditional curriculum materials for K–12 [ 65 ]. It performed a qualitative analysis on the design aspects and evaluation for AR Learning Environments (ARLES). Its focus was to investigate ARLES designed for kindergarten and primary and/or secondary school, as well as to explore learning theories as basis for effective learning experiences. They found out that there are three inherent AR affordances to educational settings: real-world annotation, contextual visualization, and vision-haptic visualization [ 65 ]. These affordances were supported by existing theories. Authors discovered that aside from the performance of students in pre- and post-tests, other aspects of the learning experience such as motivation and satisfaction were usually observed.

However, it can be noted that the aforementioned paper focuses only on K–12 education. Our paper will focus on different target groups of the AR applications evaluated.

As this research area matures and the use of AR in education grows, it is important to analyze its impact appropriately to have relevant and valid feedback for the stakeholders involved in the process. Thus, this paper presents a systematic review on how studies have been evaluating AR in education.

The contributions of this paper are:

The use of a robust research methodology to collect and analyze papers that perform educational evaluation of AR educational applications (“ Methodology ” section)

A classification and discussion of studies that evaluate educational aspects of such AR systems (“ Results and discussion ” section)

Guidelines to evaluate educational aspects of AR applications (“ Guidelines for educational evaluation ” section)

Methodology

Considering the complexity of the educational field, such as different learning needs and times, to name a few, and its implications for technology acceptance and use, a systematic review was conducted to investigate how researchers are evaluating their AR systems. This review followed the PRISMA protocol [ 57 ] as shown in Fig.  2 .

PRISMA protocol diagram

Research questions

Our main question was “how do researchers evaluate AR-based educational technology?”. To guide data extraction, analysis, and synthesis, sub-questions were formulated as listed below. The questions are divided into three categories: descriptive, classificatory, and relation and effect.

Descriptive questions :

What is the evolution in number and type of research from 2009 to 2017?

What institutions are most involved in performing this type of research?

Classificatory :

What are the different designs (methodologies) used in these studies?

What are the target populations used in these studies?

What are the constructs being analyzed?

What are the domains of the different applications tested?

What types of research questions are investigated?

What are the types of AR technology used?

What is the problem being analyzed?

Is the application based on any educational theory?

What technologies AR is combined with?

How was the involvement of teachers in the evaluation process?

Did the study use multiple metrics (both quantitative and qualitative)?

Did the study use multiple metrics for educational evaluation purposes?

Relation and effect :

What is the kind of impact of the tool analyzed?

Systematic review procedure

The first step was to establish the search string for paper selection. The search string was created based on our research questions. The terms were defined along with synonyms found in the literature as shown in Table  1 .

Then, the databases for the search were defined. Papers were searched automatically in three databases: ACM, IEEE Xplore, and Science Direct. Also, papers were searched in the main Brazilian Conference related to Informatics in Education, the Brazilian Symposium on Informatics in Education, SBIE. This search was performed manually in the Google Scholar platform using our search string.

The automatic search was performed in the databases using the same open-source paper crawler software that was used by Roberto et al. [ 63 ]. This crawler enabled authors to automate the process of retrieving papers. It uses only the search string as input, and it accesses the digital libraries to search in the title, abstract, and keywords of each paper. The crawler collects the papers, eliminates duplicate versions, and creates a spreadsheet containing all the works with their title, year, source, primary affiliation, abstract, and web address.

For papers to be included in the study, they must meet the following criteria:

Papers published in English with more than four pages

Papers were only considered once (in case of repetitive papers, we considered the more complete or the most recent one)

Papers published from 2009 to 2017

Papers that explicitly mentioned their evaluation methodology

The papers must have at least an AR prototype working

The AR solution must be tested with its end users

The solutions presented must be applied to learning a new concept or skill

Papers that intended to evaluate learning aspects

First, a search was performed in the databases using the search strings. Then, in the pre-selection phase, the researchers screened the papers by reading their title, abstract, and conclusion to eliminate the ones clearly not related to the research question. Later, we applied the inclusion criteria to those papers. These papers were screened to evaluate their quality concerning quantitative and qualitative aspects. In the extraction phase, we read the papers to extract relevant data concerning the research questions.

Data extraction

We extracted relevant information from the selected papers as listed below. The data was organized in a spreadsheet.

University/research group

Source (conference or journal)

Methodology design

Target population

Application domain

Type of research question

Implications for practice

Type of AR technology (tracking, display, interaction)

What constructs does it evaluate?

Is the application based on educational research?

What are the implications of the findings in research and practice?

What is the impact of the tool analyzed (positive or negative)?

Observations

Quality criteria evaluation

The QualSyst standard was used as a guideline for quality control [ 40 ]. This questionnaire consisted of 14 items evaluating study questions concerning design methodology, sample, outcomes, results outcomes, description, and conclusions. Some items were not scored due to their non-applicability in the study’s methodology (e.g., evaluator and user blinding); in these cases, we used n/a (not applicable) in the table. Other items such as interventional and random allocation were applied only in some cases. Each item was graded as it fulfilled the requirements in three categories: total, partial, and none with assigned scores of 2, 1, or 0, respectively. The total sum was divided by the maximum possible points (e.g., 10 items × 2 points = 20 points). The final score of each paper formed a grade. In case the paper did not conduct one type of research, qualitative or quantitative, a dash was used (-) to represent this situation in the spreadsheet.

Threats to valitidy

Authors are aware of the importance of considering threats to validity in order to judge the systematic review strengths and limitations. The main issues in this type of research are related to incomplete sets of relevant papers and researcher bias regarding quality analysis.

Limitations with search string, scientific databases, and search strategy can result in an incomplete set of relevant papers. As a way to mitigate the risks, the following strategies were used: first, in order to validate the search string, the terms were discussed among the authors. The authors have a different set of skills, two of them hold Ph.D. degrees in the field of Computer Science, one has a Ph.D. degree in Education, and one has a B.A. in Languages and is currently a Ph.D. candidate in Computer Science. All of them are teachers with experience in different educational levels, from early childhood education to post-graduate education. Second, the scientific databases that publish works from the most important conferences and journals in the area were selected, along with the papers published in the main Brazilian conference in the area. Third, the crawler uses a different approach to maximize the number of papers found. Instead of using the complete search string, eight different searches were performed using the combination of every term in both parts of the search string, which increases the number of papers collected [ 63 ].

The qualitative analysis of the papers was conducted by one of the authors. Since this may lead to a researcher bias, 15% of the papers were randomly selected to compose a set of control papers in order to increase credibility. The other authors examined the control papers to analyze them concerning their quality. The authors compared their results using Cohen’s Kappa coefficient, which measures the agreement between the two classifications taking into account how much agreement would be expected to be present by chance [ 11 ]. The coefficient lies between −1.0 and 1.0 in which 1.0 denotes perfect agreement, 0.0 indicates that any agreement is due to chance, and negative values present agreement less than chance. There is no consensus on what are good levels of agreement. Nevertheless, studies [ 1 ] mention that there is no agreement for negative values, poor agreement between 0.00 and 0.20, fair agreement between 0.21 and 0.40, moderate agreement between 0.41 and 0.60, good agreement between 0.61 and 0.80, and very good agreement for values higher than 0.80. In our work, the qualitative analysis Cohen’s Kappa was 0.7969, which is close to very good agreement among the authors.

Results and discussion

This section describes and discusses the results of the systematic review.

The search in the databases using the search strings returned 607 articles, and 148 papers remained after the pre-selection phase. Finally, after applying the inclusion criteria, 45 papers were eligible for this study. The results from all the papers are available in an online database, which can be collaboratively updated Footnote 2 .

Quality of report

The quantitative and qualitative assessments are available at Appendixes A.1 and A.2 , respectively.

Descriptive questions

Questions 1 and 2 are in this category. Figure  3 shows that although no research was found in 2009, the research in this field is steadily growing, reaching the highest number of papers in 2014. Although the number of papers per year has decreased compared to 2014, we observe that the interest in evaluating AR for education remains.

Papers according to the year of publication

Table  2 presents the institutions involved in the research.

Table  3 shows the venues where the studies have been published.

Figure  4 evidences that the methodology most commonly used is the experimental design, while the quasi-experimental design appeared in fourth place. The essential feature of experimental research is that the researcher deliberately controls and manipulates the conditions, which determine the events of interest [ 12 ]. Quasi-experiments are used when subjects must be allowed to choose their treatment, which is the main difference when compared to experimental designs.

Papers according to the design methodology

Questionnaires were the second most popular method among the studies. They consist in a series of questions or prompts aimed at gathering information from subjects. The questionnaires used in the papers were designed in different ways and for varied purposes. As examples, Zhang et al. [ 82 ] used a questionnaire to investigate flow experience and Wei et al. [ 77 ] to assess creative design learning motivation. In turn, Ibánez et al. [ 36 ] designed an open-ended questionnaire. Regardless of its structure and aim, Cohen et al. [ 12 ] point out that an ideal questionnaire must be clear and unambiguous.

Observations appeared in seven studies while only one work reported a case study. Merriam [ 56 ] explains that observations take place where a given phenomenon naturally occurs. She points out that the skills to be a good observer must be learned; thus, training and mental preparation is important. She highlights the need to define what to observe as well as to write careful and useful field notes.

Case study, on the other hand, is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident [ 81 ]. Merriam [ 56 ] points out that the most defining characteristic of a case study lies in delimiting the case to be studied. Thus, case study research uses purposive sampling rather than random sampling [ 25 ].

It is important to highlight that a high number of papers (32, total) reported a combination of methods or metrics. The most common combination is the experiment coupled with questionnaires. However, these multiple metrics usually not only evaluated education, but also other aspects such as motivation and satisfaction. The results for this question also evidenced a predominance of quantitative methods in the works.

Question 4 refers to the target population of the studies as seen in Fig.  5 .

Papers according to the target population

Figure  5 shows that the most popular target audience are undergraduate students and elementary school children. High school students appeared in seven works, thus being the third most popular audience for AR tools.

Other groups were also considered in these papers. For instance, two papers targeted general audiences of users. For instance, in Sommerauer and Müller [ 67 ], the population was an exhibition audience, which included heterogeneous genders, age groups, and educational levels. These varied target populations show that AR can expand the barriers of the school setting and achieve both formal and informal environments.

Also, parents were the audience in Cheng and Tsai [ 10 ]. This paper also targets children in both elementary and preschool. Tobar-Muñoz et al. [ 71 ] present an AR tool for children with varied ages and special needs. Finally, four papers are targeted to workers in different fields, such as engineering [ 8 ] and surgery [ 45 ]. This data evidence that AR can also be successfully used for training.

Hence, data show that although there has been a preference for undergraduate students and elementary school children, AR can be used by a variety of people, with different needs and in different contexts.

Question 5 was about the constructs evaluated in the studies as displayed in Fig.  6 .

Papers according to the constructs evaluated

Figure  6 reveals that many studies did not evaluate solely educational aspects. Twelve works evaluated more than one aspect. The majority of the papers evaluated knowledge retention or performance.

Some applications were under development or had been recently developed; thus, usability aspects, such as users’ attitudes and satisfaction, were also analyzed. Martín-Gutiérrez et al. [ 52 ] point out that the study was carried out with the beta version of the tool, which was tested with 235 students. These authors, thus, also evaluated user’s satisfaction. In turn, Tarng et al. [ 70 ] investigated the attitudes of experimental group students after using the AR system. The authors explain that the questions in their study were categorized in learning contents, interface design, and applications.

Behavior and motivation were also evaluated in eight studies. Other studies evaluated constructs related to the theories they used, such as flow experience [ 8 , 36 ] and dimensions of learning style [ 49 ].

Other aspects evaluated were creativity [ 77 ], teaching effects [ 77 ], and learner’s opinions [ 69 ]. This variety evidences that due to the complexity of the learning environment, different aspects can be the focus of educational or learning evaluation. Depending on the focus of the studies, such as training, authors would focus on more mechanical aspects such as precise skills development and time. Conversely, studies focusing on the school environment may focus their attention on the role of the teacher, flow experience, or student’s motivation to learn.

Question 6 concerns the application domains of knowledge as shown in Fig.  7 .

Papers according to their domains of knowledge

Figure  7 shows that most AR tools are related to STEM fields. STEM is an acronym that refers to the fields of science, technology, engineering, and mathematics. The second most popular domain for applications are humanities, followed by medicine and health.

Question 7 investigated the types of research questions in the works. The questions were classified according to their types as proposed by Easterbrook et al. [ 25 ]. These authors divide research questions in two types: design questions, which are usually asked by software engineers in order to better ways to do software engineering, and knowledge questions, which are described below:

Exploratory questions: are asked in the early stages of research when researchers are attempting to understand the phenomena, e.g., existence questions, description and classification, descriptive comparative

Base-rate questions: are frequently asked after having a clearer understanding of the phenomena. They might be frequency and distribution questions and descriptive process

Relationship questions: are meant to understand the relationship between two different phenomena

Causality questions: are an attempt to explain why a relationship holds and identify its cause and effect, e.g., causality questions, causality-comparative questions and causality-comparative-interaction questions.

Figure  8 presents the types of research questions found in the papers.

Papers according to their research questions

Twenty-three papers asked more than one question. The chart shows that the majority of the papers asked relationship questions; those papers aimed to describe the effect of AR compared to other resources and its relationship with different aspects (e.g., academic achievement or motivation). The second most common type of question was exploratory ones, mainly descriptive comparative (present in 19 papers). Design questions were asked by two studies and causality ones by one study. This amount of exploratory questions may indicate that research in the use AR tools for education might still be in early stages, in which researchers attempt to better understand the field and the implications of such technology in education. Also, they want to understand what are the better ways to develop their tools, as evidenced in the design questions.

AR technologies used in the studies were classified according to their tracking, display, and interaction techniques. As concerns the displays used, Fig.  9 shows that screen-based and handheld were the most frequent used displays (21 and 14 papers, respectively). Screenbased displays are known for their cost-efficiency since they require off-the-shelf hardware and standard PC equipment. They are also largely present in schools nowadays and were usually well evaluated by users.

Papers according to the display used

On the other hand, the popularization and technical advancements in smartphones make handheld displays a good option for AR applications. These devices are minimally intrusive and highly mobile [ 83 ]. They enable high flexibility, as shown in Jerry and Aaron [ 38 ], in which a context-aware AR learning solution is proposed as a scaffolding platform for outdoor field learning. Tarng et al. [ 70 ] used these displays to provide situated learning.

Two papers used head-attached displays. Although these displays provide a better field of view, fashion constraints are a common issue. For instance, Martín-Gutiérrez et al. [ 52 ] reported that the HMD use was not comfortable. The cables linking the glass and camera with the PC interfered with user’s movement.

Five studies presented spatial displays. Three studies did not provide enough information about the system evaluated; therefore, it was not possible to classify these systems in all three categories of AR [ 27 , 28 , 38 ].

As regards to tracking, 35 papers presented vision-based tracking and five papers presented sensor-based tracking as shown in Fig.  10 . Two papers presented hybrid tracking.

Papers according to the tracking technique

Vision-based tracking can be divided in two categories, marker-based and markerless as illustrated in Fig.  10 .

Marker-based was the most common type found (25 works). It is a very popular choice since there are many marker-based kits available for a low cost. Most papers presented positive outcomes regarding these tools. However, markers can be intrusive in the scene.

On the other hand, markerless systems do not require the use of markers. In this case, the environment itself acts as a marker. It allows guidance information to be superimposed on a real game board, for example. This type was chosen in ten studies.

Finally, as regards to interaction techniques, 19 papers presented a more traditional type of interaction using buttons, touch, or simply providing visualization of the augmented content. Their use was generally positive.

The second most common choice was tangible interaction (13 papers). These interfaces are promising as they take advantage of the familiarity of everyday objects to ease the interaction. Their use provided positive results.

Haptic interfaces were chosen in four papers. One paper presented collaborative interaction [ 48 ]. No papers chose hybrid interfaces.

Figure  11 shows the interaction techniques used in the papers.

Papers according to the interaction techniques

Question 11 investigated if the studies were based on any educational theory as presented in Table  4 .

Table  4 evidences that most papers mentioned educational theories. However, 19 studies did not mention any theory. The most mentioned theories were situated learning theory and the cognitive theory of multimedia learning and cognitive load theory, mentioned in three works each. The situated learning theory emphasizes the reality of learning activities; thus, the context in which the activity naturally occurs is indispensable. AR allows real-life experiences to be enhanced with virtual content, hence expanding learning horizons.

In turn, the cognitive theory of multimedia learning (CTML) states that people learn better from pictures and words rather than pictures alone. This theory is based on three assumptions: (a) people possess two channels for processing information (the auditory/verbal and visual/pictorial), (b) there is a limited amount of information each channel can process at a time, and (c) learning is an active process of selecting relevant information, organizing them into coherent mental representations, and finally integrating those representations with existing knowledge [ 54 ].

Inquiry-based learning was mentioned in two studies. As an example, Jerry and Aaron [ 38 ] mentioned this theory, which is an approach to teaching and learning that places students’ questions, ideas, and observations at the center of the learning experience [ 60 ]. Hutchings [ 34 ] adds that the process of inquiry is in the ownership of the learners; thus, inquiry-based learning is fundamentally concerned with establishing the context within which inquiry may best be stimulated and students can take charge of their learning.

Mobile learning was mentioned in two works. Mobile learning or, simply, m-learning is the didactic-pedagogical expression used to designate a new educational “paradigm” based on the use of mobile technologies [ 58 ]. Also, McGreal [ 55 ] adds that “m-learning happens in context in which it is needed and relevant and is situated within the active cognitive processes of individual and groups of learners.” Thus, it takes advantage of the widely available mobile devices to provide access to learning anywhere and anytime, which changes many paradigms of traditional education.

The learning styles theory was also found in two papers. For instance, Zhang et al. [ 82 ] was based, specifically, on the kinesthetic learning style theory. Learning styles are the general approaches used by students in learning a new subject [ 61 ]. These “overall patterns” that generally direct learning behavior are divided in dimensions, for example, the sensory preference [ 14 ]. Sensory preferences can be divided into four main areas: visual, auditory, kinesthetic (movement-oriented)—explored in Zhang et al. [ 82 ], and tactile (touch-oriented) [ 61 ].

Studio-based learning theory was found only in one study [ 77 ]. It is a learning model first developed as part of education and training and later adopted by architectural education in the 1800s [ 43 ]. This model has its roots on the notion of the apprentice in the atelier where they worked and learned skills of the master design or artist. Young apprentices did not learn in isolated schools, but were exposed to real adult world and worked on real products in the community.

Other theories were represented by one paper each. Another one was the flow theory that brings the concept of flow which is a state of complete absorption or engagement in an activity that acts as a motivating factor in daily activities such as work, sport, and education [ 16 ]. This state encourages a person to persist at an activity due to experience rewards it promises, and it fosters the growth of skills over time [ 59 ].

Most of these theories have in common a learner-centered approach, thus focusing more on student’s discovery, construction and interaction process, and the attachment to the context of learning. In this sense, AR, along with other types of technology, can expand the learning horizons. Some theories focus on understanding learning processes to provide a more effective experience for the students considering their personal needs and abilities. As shown, the trend is to look at AR instructional design from the learners’ perspective.

The following question was: “what technologies AR is combined with?”. This question inquired if AR applications were combined with other technologies and what kinds of technologies they were combined with. As can be seen in Table  5 , 37 papers did not combine AR with other types of technology. The other papers combined it with different types of technology, such as YouTube tutorial, personal blogs, digital sketching, notes and texts provided by the teacher, robotics, mobile pedestrian navigation, virtual reality and digital sketching using hybrid models (DS/HM), and web-based simulation environment. All these technologies appeared one time each. Although it is evident a preference to not combine AR with other types of technology, it is interesting to note that in the classroom environment, AR is another possibility among many others already present in that environment. It is helpful, thus, to understand how these multiple possibilities can work together to scaffold learning.

Question 13 refers to the involvement of teachers in the evaluation process as shown in Fig.  12 .

Papers according to the involvement of teachers

Most studies did not involved the teachers in the studies. Some of the studies were in different contexts, such as library instruction by Wang et al. [ 74 ]; thus, in this case, authors mentioned the role of the librarian.

Nevertheless, 13 studies reported the involvement of teachers in different ways and levels. Figure  12 evidences that the teacher may be involved in the design and evaluation process of AR educational tools in different ways. The most common way was the teacher(s), or in some cases, schools directors, working as consultants or curators. Teachers were consulted for different purposes, such as problematic contents to teach [ 82 ] or to review or modify tests [ 36 , 67 , 82 ].

Seven studies involved the teachers as evaluators of student outputs. As an example, [ 44 ] explains that “AR will be used for self-assessment and that the teacher can mark the answers and give the scores on internet web page.”

Another role was to act as a tutor (six mentions). That means the teachers had a role of explaining content to students or monitor their work. For instance, [ 73 ] mentions that “the procedure of experiment is started with teacher lectures to all students in class.”

Also, five papers reported the participation of the teachers as creators of learning experiences. Cubillo et al. [ 17 ] reports that the teachers can follow an established procedure to create content using the tool. In da Silva et al. [ 19 ] and da Silva et al. [ 20 ], teachers were not able to create the applications by themselves since the AR tool evaluated needs an authoring tool, but they were able to design the activities to be worked and programmers created the content accordingly.

Finally, in Frank and Kapila [ 30 ], the teacher was considered a confounding, as illustrated in these lines: “teacher’s feedback was prevented in the design of the experiment by having student participants tested individually, being directed to perform the activity immediately after the pre-assessment and then immediately to complete the post-assessment.”

The results for questions 14 and 15 can be seen in Fig.  13 . Q14 refers to the use of multiple metrics. We can see that 24 studies used both quantitative and qualitative metrics and that 21 did not adopt this practice. However, most papers did not use both metrics to evaluate learning gains.

Papers according to the use of multiple metrics

Papers, such as Zhang et al. [ 82 ] and Wei et al. [ 77 ] used both types of metrics to evaluate learning gains. Zhang et al. [ 82 ] investigated the application of location-based AR to astronomical observation instruction. It used both quantitative and qualitative data to investigate aspects related to learning. To gather qualitative data, the authors performed an interview with teachers to understand the limitations of traditional teaching methods as a reference for the system’s design proposed. The quantitative data assessed learning effectiveness and motivation.

On the other hand, Wei et al. [ 77 ] showed a general technical creative design teaching scheme that includes AR. It used questionnaires to assess creative design learning, motivation, and teaching efficiency. There were also tests on creative design learning motivation, teaching effects, and creativity of the output.

This is an interesting aspect since the educational aspects are very complex and only quantitative metrics are not enough to understand the nuances involved in the process.

Relation and effect questions

Question 15 was in this category. This question explores the kind of impact of the tools analyzed in the studies. As shown in Fig.  14 , 33 papers reported positive of the results. For instance, Jerry and Aaron [ 38 ] proposed a system that promoted a better relation to physics concepts.

Papers according to the AR impact in education

Ibánez et al. [ 36 ] revealed that the AR-based application was more effective than the web-based one in promoting student’s knowledge. The four teachers in Wei et al. [ 77 ] considered the creative designs produced with AR by students more novel, sophisticated, and with more practical value.

In terms of performance improvement, Yeo et al. [ 80 ] reported that the AR image overlay and laser guidance improved the training process of needle placement. The participants who trained with overlay guidance performed better even when required to do freehand insertions. Zhang et al. [ 82 ] describe that in outdoor teaching environments, altering tool factors significantly enhances performance factors.

Regarding usability aspects, Wei et al. [ 77 ] reported that students considered the teaching contents with AR relevant and so had greater satisfaction.

The systems were described as convenient/interesting in some studies. Additionally, students in Tarng et al. [ 70 ] considered the virtual scenes and butterflies very realistic, and they would like to use it again in the future.

Reduction in costs were also reported. Student’s attention was also significantly improved due to the introduction of AR technology as reported in Wei et al. [ 77 ].

AR also enabled learning formal contents in informal environments as shown in Sommerauer and Müller [ 67 ]. This study pointed out that the empirical evidence suggests that AR has the potential to be an effective tool for learning mathematics in a museum. Students also perceived AR as a valuable add-on of the exhibition.

Eleven papers reported mixed results. That means the results could be either positive or negative for one aspect and neutral for others, for example. This situation is illustrated in Martín-Gutiérrez et al. [ 52 ]. This paper reported improvement on user’s spatial skills while working on their own; the statistic results show that use of the HMD device does not provide any difference when obtaining spatial ability upgrades with respect to the PC monitor. Authors argue that this result may be caused by the fact that HMD use is not the most suitable as users stated that the glass and camera set were not comfortable.

In Wang et al. [ 74 ], the proposed librarian system was more helpful in promoting the learning performance of learners with the field-dependent cognitive style than the conventional librarian instruction, particularly for learning content associated with application and comprehension.

Chen and Tsai [ 9 ] revealed that there was no gender difference in learning. This study investigated the AR’s impact depending on student’s personal learning styles (there was an impact) and personal gaming skills (there was no impact). Chen and Tsai [ 9 ] revealed a neutral outcome.

Another example is [ 37 ], which reported positive regarding intrinsic motivation, but slightly negative (although not significantly different) regarding selflearning. Nevertheless, no paper reported only negative or neutral outcomes.

Guidelines for educational evaluation

Through this literature review, authors were able to understand the current status of AR evaluation in education. In this section, we will discuss some principles that are important to be taken into account in similar situations. These aspects have already been discussed in [ 18 ].

Many studies have pointed out the importance of multiple metrics in research design. For instance, Easterbrook et al. [ 25 ] point out its usefulness and highlight the importance of employing both quantitative and qualitative metrics as a way of compensating the weakness of each method. Cohen et al. [ 12 ] explain that there are many advantages of using multimethod approaches in social research. The authors highlight two of them:

While single observation in fields such as physics and chemistry usually yield sufficient and unambiguous information, it provides a limited view of the complexity of human behavior and interactions.

Exclusive reliance on one method may bias or distort the researcher’s picture of a particular reality he/she is investigating.

Although not all the papers used multiple metrics to evaluate educational aspects, we observed that many papers did use them in their studies.

Another important issue is technology integration into the classrooms. In order to effectively evaluate new educational technology, it is important to effectively integrate them in the schools. Dexter [ 21 ] points out two premises for effective integration and implementation of technology for K–12 classrooms, that are:

The teacher must act as an instructional designer, planning the use of technology to support learning.

Schools must support teachers in this role.

It is important for researchers and developers to have an understanding on how teachers will integrate new technologies into their lessons since this will shape student’s learning opportunities. Fitzpatrick [ 26 ] stresses the need to involve teachers in the process of adopting new technology, so the activities are integrated to their lesson plan and meaningful to the students. For instance, activity theory [ 47 ] shows that activities are culturally mediated and inserted into a given context that includes the mediation of artifacts, of the community, and of its rules and its division of labor. In the process of transforming the activity of teaching into learning, there is a whole complex of mediations involving the curriculum, the educational rules, teacher’s training, and artifacts to name a few. This complex scenario needs to be taken into account in order for researchers to understand the changes caused by the introduction of a new artifact and the changes needed to expand and adjust the system.

Hence, taking this information into account, it is possible to infer that teachers need to have a very active approach when it comes to use and evaluation of technology in education. However, the data showed that only five papers considered the teacher as a creator in their evaluation process.

Crompton [ 15 ] explains that the evaluation of a piece of technology in isolation will tend to focus on various aspects of the technology itself, such as screen design and text layout. On the other hand, the evaluation of a courseware within the course itself will allow for examination of other factors that will lead to successful integration of the product within the course. Some of these aspects are:

Educational setting

Aims and objectives of the course

Teaching approach

Learning strategies

Assessment methods

Implementation strategy

Formative evaluations as stated by Scriven are typically conducted during the development or improvement of a program, person, or product, and it is conducted with the intent to improve [ 66 ]. On the other hand, summative evaluation is typically quantitative, using numeric scores or letter grades to assess learner achievement. Thus, a comprehensive evaluation involving both types of assessment is advisable in order to have a better overview of the process and its outcome.

Final remarks

Through this research, we identified AR’s potential to be applied in learning contexts. Developments in AR technology have enabled researchers to develop and evaluate more tools in the field of education. Hence, it was evident a growing interest in evaluating its impact in the learning process.

Results have shown that most studies combined different methodologies to evaluate their tools; however, only few papers combined them to evaluate educational gains.

Most of these papers used multiple metrics but to evaluate different aspects rather than just learning, such as usability and efficiency. Merriam [ 56 ] explains that all research designs can be discussed in terms of their relative strengths and limitations. She claims that their merits are related to select the most appropriate ones to address the research problem. Cohen et al. [ 12 ] argue that there are many advantages of using multimethod approach in social research. They highlight that (a) while single observation in fields such as physics usually yield sufficient and unambiguous information, it provides a limited view of the complexity of human behavior and interactions, and (b) exclusive reliance on one method may bias or distort the researcher’s picture of a particular reality.

It was also evident that most studies did not involve the teacher as an instructional designer. However, teachers were involved in many studies in a wide range of ways from consultant to creator. Fitzpatrick [ 26 ] highlights the need to involve teachers in the process of adopting new technological tools, so activities are integrated into their lesson plans and, thus, meaningful to the students.

Although AR has been shown to be helpful for teachers, it can also be inferred that its use, in some situations, may decrease the role of the teacher as the only source of knowledge since it may enable learners to be aided by other peers, trainers, or even their parents depending on the situation.

In this review, we noticed that there are solutions being developed to different age groups and knowledge domains. However, it was noticed a lack of evaluation of AR systems aimed at very young learners.

Regarding the types of questions asked, most papers presented more than one question. These questions were mainly relationship and descriptive-comparative ones. Those papers intended to describe the effect of a given AR technology comparing it with different resources as well as its relationship with different aspects, such as academic achievement or motivation, which indicates that the field is still maturing when it comes to evaluating AR educational impacts.

The papers were also classified according to the tracking, display, and interaction techniques used. It was noticeable that this choice of technology varied deeply depending on the learning objectives of the tool. However, this choice had an impact in the possibilities and limitations of use of the applications.

We also investigated if the papers based their work in any educational theory. Most papers mentioned educational theories. However, 19 studies did not mention any theory. It is important to highlight that educational theories may help to unravel contributions of AR tools as well as its limitations. In addition, it may help to understand how AR unique features may impact in the learning setting. The theories mentioned varied considerably, but something that most of them had in common is a learner-centered approach, thus putting the focus on student’s discovery, construction, and interaction processes and the attachment to the learning context.

It is noticeable that AR can expand the learning horizons. Some of the theories focus on understanding learning processes to provide a more effective experience for students considering their personal needs and abilities. We observed the need to look at AR instructional design from the perspective and limitations of the learners themselves.

The latter question investigated the kinds of impact of the results of the studies. Most of them presented positive outcomes. AR has been proved to be a helpful tool concerning many aspects of learning. In this sense, studies presented positive outcomes regarding a wide range of aspects, such as learning, academic performance, and motivation, among others.

Neutral outcomes were also reported as in some studies; the proposed AR system generated equivalent learning performance when compared to a traditional one. However, as already discussed, in many cases, results were neutral for one aspect and positive for others.

The analysis evidenced that AR can help to promote independence and interest among students, which can lead to more student-centered approaches, in which students are the center of their own learning and may apply it in more practical ways. The use of AR also enabled students to experience more concrete situated learning experiences, and together with mobile technologies, it may help to extend learning to different environments in a contextualized way, such as museums and student’s campi.

To sum up, during this review, it was noticed that AR has unique affordances that can impact the learning experience. As technology matures, researchers are increasingly concerned with how to incorporate real classroom/learning issues into their investigation.

Thus, authors discussed some guidelines for AR educational evaluation based on the lessons learned. First, based on the literature review, we advocate for the use of multiple metrics both quantitative and qualitative in order to have a better overview of the technology inserted in the teaching context as well as its effects.

Second, although it is not always possible to have a longitudinal evaluation, it is recommended to have a comprehension of more than punctual assessments but rather understand its effect in student’s development in a longer term. Finally, as it is widely recognized that teachers play a major role in technology adoption in the schools, we advocate for the involvement of teachers in the evaluation in more active ways as possible. Moreover, it is important to have tools that are flexible enough in order to facilitate teachers’ and students’ input of content.

As for limitations, due to the limited number of databases used, authors are aware that results may not fully represent the research development in the field.

Implications of the research

As implications of this research, it was noticed the need for more authoring tools that would enable users to create their own materials independently. Moreover, it is evident the need for more research regarding the evaluation of AR, especially, long-term ones since they could provide a better overview of the process of using this technology into the learning environment.

A.1 Quantitative criteria

Table 6 shows the scores of the quantitative evaluation of each paper.

Question/objective sufficiently described?

Study design evident and appropriate?

Method of subject/comparison group selection or source of information/input variables described and appropriate?

Subject (and comparison group, if applicable) characteristics sufficiently described?

If interventional and random allocation was possible, was it described?

If interventional and blinding of investigators was possible, was it reported?

If interventional and blinding of subjects was possible, was it reported?

Outcome and (if applicable) exposure measure(s) well defined and robust to measurement/misclassification bias means of assessment reported?

Sample size appropriate?

Analytic methods described/justified and appropriate?

Some estimate of variance is reported for the main results?

Controlled for confounding?

Results reported in sufficient detail?

Conclusions supported by the results?

A.2 Qualitative criteria

Table 7 shows the scores of the quantitative evaluation of each paper.

Context for the study clear?

Connection to a theoretical framework/wider body of knowledge?

Sampling strategy described, relevant, and justified?

Data collection methods clearly described and systematic?

Data analysis clearly described and systematic?

Use of verification procedure(s) to establish credibility?

Reflexivity of the account?

https://www.instructionaldesigncentral.com/whatisinstructionaldesign

The database with the selected papers is available at: https://papercatalog.000webhostapp.com

Altman D (1990) Practical statistics for medical research. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, London. ISBN 9780412276309. https://books.google.com.br/books?id=v-walRnRxWQC .

Google Scholar  

Amor B (2014) Experience art through relative realities with Samsung digital gallery at Yuchengco Museum. https://goo.gl/KLygPJ . Accessed 2 Oct 2018.

Azuma RT (1997) A survey of augmented reality. Presence: Teleoper. Virtual Environ 6(4):355–385. ISSN 1054-7460. https://doi.org/10.1162/pres.1997.6.4.355 .

Balog A, Pribeanu C (2010) The role of perceived enjoyment in the students’ acceptance of an augmented reality teaching platform: a structural equation modelling approach. Stud Inform Control 19(3):319–330.

Article   Google Scholar  

Billinghurst M, Duenser A (2012) Augmented reality in the classroom. Computer 45(7):56–63. ISSN 0018-9162. https://doi.org/10.1109/MC.2012.111 .

Billinghurst M, Clark A, Lee G, et al (2015) A survey of augmented reality. Found Trends Ⓡ Hum–Comput Interact 8(2-3):73–272.

Bosque LD, Martinez R, Torres JL (2015) Decreasing failure in programming subject with augmented reality tool. Procedia Comput Sci 75:221–225. ISSN 1877-0509. https://doi.org/10.1016/j.procs.2015.12.241 . http://www.sciencedirect.com/science/article/pii/S1877050915037023 .

Chang K-E, Chang C-T, Hou H-T, Sung Y-T, Chao H-L, Lee C-M (2014) Development and behavioral pattern analysis of a mobile guide system with augmented reality for painting appreciation instruction in an art museum. Comput Educ 71:185–197. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2013.09.022 . http://www.sciencedirect.com/science/article/pii/S0360131513002868 .

Chen C-M, Tsai Y-N (2012) Interactive augmented reality system for enhancing library instruction in elementary schools. Comput Educ 59(2):638–652. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2012.03.001 . http://www.sciencedirect.com/science/article/pii/S0360131512000589 .

Cheng K-H, Tsai C-C (2014) Children and parents’ reading of an augmented reality picture book: analyses of behavioral patterns and cognitive attainment. Comput Educ 72:302–312. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2013.12.003 . http://www.sciencedirect.com/science/article/pii/S0360131513003278 .

Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46.

Cohen L, Manion L, Morrison K (2003) Research methods in education. Taylor & Francis ebook collection. Taylor & Francis. ISBN 9780203224342. https://books.google.co.uk/books?id=5twk1pHwyL8C . Accessed 2 Oct 2018.

Contero M, Gomis JM, Naya F, Albert F, Martin-Gutierrez J (2012) Development of an augmented reality based remedial course to improve the spatial ability of engineering students In: 2012 Frontiers in Education Conference Proceedings, 1–5. https://doi.org/10.1109/FIE.2012.6462312 .

Cornett CE (1983) What you should know about teaching and learning styles. Fastback Series In: Phi Delta Kappa Educational Foundation. ISBN 9780873671910. https://books.google.at/books?id=SlckAQAAMAAJ . Accessed 2 Oct 2018.

Crompton P (1996) Evaluation: a practical guide to methods In: Learn Technol Dissemination Initiative, 66.

Csikszentmihalyi M (1990) Flow: the psychology of optimal experience, 6 edn. Harper & Row, New York.

Cubillo J, Martín S, Castro M, Diaz G, Colmenar A, Boticki I (2014) A learning environment for augmented reality mobile learning In: 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, 1–8. https://doi.org/10.1109/FIE.2014.7044039 .

da Silva MMO, Roberto RA, Teichrieb V, Cavalcante PS (2016) Towards the development of guidelines for educational evaluation of augmented reality tools In: 2016 IEEE Virtual Reality Workshop on K-12 Embodied Learning through Virtual Augmented Reality (KELVAR), 17–21. https://doi.org/10.1109/KELVAR.2016.7563677 .

da Silva M, Roberto R, Teichrieb V (2013) Evaluating an educational system based on projective augmented reality In: Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), vol 24, 214.. Sociedade Brasileira de Computação, Porto Alegre.

da Silva MMO, Roberto R, Teichrieb V (2015) Evaluation of augmented reality technology in the English language field In: Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), vol 26, 577.. Sociedade Brasileira de Computação, Porto Alegre.

Dexter S (2002) eTIPS-Educational technology integration and implementation principles. Designing Instr Technol-Enhanc Learn 24:56–70.

Dünser A, Billinghurst M (2011) Evaluating augmented reality systems, 289–307.. Springer New York, New York. ISBN 978-1-4614-0064-6. https://doi.org/10.1007/978-1-4614-0064-6_13 .

Dünser A, Grasset R, Seichter H, Billinghurst M (2007) Applying HCI principles to AR systems design. Technical report. HIT Lab NZ.

Dünser A, Grasset R, Billinghurst M (2008) A survey of evaluation techniques used in augmented reality studies In: ACM SIGGRAPH ASIA 2008 courses, SIGGRAPH Asia ’08, 1–27.. ACM, New York. https://doi.org/10.1145/1508044.1508049 . http://portal.acm.org/citation.cfm?id=1508044.1508049 .

Easterbrook S, Singer J, Storey M-A, Damian D (2008) Guide to advanced empirical software engineering, chapter Selecting Empirical Methods for Software Engineering Research. Springer London, London. ISBN 978-1-84800-044-5. https://doi.org/10.1007/978-1-84800-044-5_11 . http://doi.org/10.1007/978-1-84800-044-5_11 .

Fitzpatrick A (2004) Analytical survey information and communication technologies in the teaching and learning of foreign languages: state-of-the-art, needs and perspectives. unesco institute for information technologies in education. Technical report. UNESCO, Moscow, Russia.

Fonseca D, Villagrasa S, Vails F, Redondo E, Climent A, Vicent L (2014) Engineering teaching methods using hybrid technologies based on the motivation and assessment of student’s profiles In: 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, 1–8. https://doi.org/10.1109/FIE.2014.7044209 .

Fonseca D, Villagrasa S, Valls F, Redondo E, Climent A, Vicent L (2014) Motivation assessment in engineering students using hybrid technologies for 3d visualization In: 2014 International Symposium on Computers in Education (SIIE), 111–116. https://doi.org/10.1109/SIIE.2014.7017714 .

Fonseca D, Valls F, Redondo E, Villagrasa S (2016) Informal interactions in 3D education: Citizenship participation and assessment of virtual urban proposals. Comput Hum Behav 55:504–518. ISSN 0747-5632. https://doi.org/10.1016/j.chb.2015.05.032 . http://www.sciencedirect.com/science/article/pii/S0747563215004033 .

Frank JA, Kapila V (2017) Mixed-reality learning environments: integrating mobile interfaces with laboratory test-beds. Comput Educ 110:88–104. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2017.02.009 . http://www.sciencedirect.com/science/article/pii/S0360131517300325 .

Hou L, Wang X (2013) A study on the benefits of augmented reality in retaining working memory in assembly tasks: a focus on differences in gender. Autom Constr 32:38–45. ISSN 0926-5805. https://doi.org/10.1016/j.autcon.2012.12.007 . http://www.sciencedirect.com/science/article/pii/S0926580512002464 .

Hsiao KF, Rashvand HF (2011) Body language and augmented reality learning environment In: 2011 Fifth FTRA International Conference on Multimedia and Ubiquitous Engineering, 246–250. https://doi.org/10.1109/MUE.2011.51 .

Hulin T, Schmirgel V, Yechiam E, Zimmermann UE, Preusche C, Pöhler G (2010) Evaluating exemplary training accelerators for programming-by-demonstration In: 19th International Symposium in Robot and Human Interactive Communication, 440–445. https://doi.org/10.1109/ROMAN.2010.5598611 .

Hutchings W (2007) Enquiry-based learning: definitions and rationale. Technical report, Centre for Excellence in Enquiry-Based. http://www.ceebl.manchester.ac.uk/resources/papers/hutchings2007_definingebl.pdf .

Ibánez MB, Di-Serio Á, Villarán-Molina D, Delgado-Kloos C (2015) Augmented reality-based simulators as discovery learning tools: an empirical study. IEEE Trans Educ 58(3):208–213. ISSN 0018-9359. https://doi.org/10.1109/TE.2014.2379712 .

Ibánez MB, Di Serio Á, Villarán D, Kloos CD (2014) Experimenting with electromagnetism using augmented reality: impact on flow student experience and educational effectiveness. Comput Educ 71:1–13. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2013.09.004 . http://www.sciencedirect.com/science/article/pii/S0360131513002571 .

Iwata T, Yamabe T, Nakajima T (2011) Augmented reality go: extending traditional game play with interactive self-learning support In: 2011 IEEE 17th International Conference on Embedded and Real-Time Computing Systems and Applications, vol 1, 105–114. https://doi.org/10.1109/RTCSA.2011.43 .

Jerry TFL, Aaron CCE (2010) The impact of augmented reality software with inquiry-based learning on students’ learning of kinematics graph In: 2010 2nd International Conference on Education Technology and Computer, vol 2, V2–1–V2–5. https://doi.org/10.1109/ICETC.2010.5529447 .

Joo-Nagata J, Abad FM, Giner JG-B, García-Peñalvo FJ (2017) Augmented reality and pedestrian navigation through its implementation in m-learning and e-learning: evaluation of an educational program in Chile. Comput Educ 111:1–17. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2017.04.003 . http://www.sciencedirect.com/science/article/pii/S0360131517300751 .

Kmet LM, Lee RC, AHFMR, Cook LS (2004) Standard quality assessment criteria for evaluating primary research papers from a variety of fields In: HTA initiative. Alberta Heritage Foundation for Medical Research. ISBN 9781896956770. https://books.google.at/books?id=UYE8HQAACAAJ . Accessed 2 Oct 2018.

Kostaras NN, Xenos MN, et al (2009) Assessing the usability of augmented reality systems In: Proceedings of the 13th Panhellenic Conference on Informatics, 197–201, Alberta Heritage Foundation for Medical Research (AHFMR). AHFMR-HTA Initiative.

Kraut B, Jeknić J (2015) Improving education experience with augmented reality (AR) In: 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), 755–760. https://doi.org/10.1109/MIPRO.2015.7160372 .

Lackney JA (1999) A history of the studio-based learning model. http://edi.msstate.edu/work/pdf/history_studio_based_learning.pdf . Accessed 2 Oct 2018.

Lai ASY, Wong CYK, Lo OCH (2015) Applying augmented reality technology to book publication business In: 2015 IEEE 12th International Conference on e-Business Engineering, 281–286. https://doi.org/10.1109/ICEBE.2015.55 .

Leblanc F, Champagne BJ, Augestad KM, Neary PC, Senagore AJ, Ellis CN, Delaney CP (2010) A comparison of human cadaver and augmented reality simulator models for straight laparoscopic colorectal skills acquisition training. J Am Coll Surg 211(2):250–255. ISSN 1072-7515. https://doi.org/http://doi.org/10.1016/j.jamcollsurg.2010.04.002 . http://www.sciencedirect.com/science/article/pii/S1072751510002735 .

Lee K (2012) Augmented reality in education and training. TechTrends 56(2):13–21. ISSN 1559-7075. https://doi.org/10.1007/s11528-012-0559-3 . http://doi.org/10.1007/s11528-012-0559-3 .

Leont’ev AN (1978) Activity, consciousness, and personality. Prentice-Hall, Englewood Cliffs. ISBN 9780130035332. http://lchc.ucsd.edu/MCA/Paper/leontev/ .

Lin T-J, Duh HB-L, Li N, Wang H-Y, Tsai C-C (2013) An investigation of learners’ collaborative knowledge construction performances and behavior patterns in an augmented reality simulation system. Comput Educ 68:314–321. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2013.05.011 . http://www.sciencedirect.com/science/article/pii/S0360131513001322 .

Mahmoudi MT, Badie K, Valipour M (2015) Assessing the role of AR-based content in improving learning performance considering Felder-Silverman learning style In: 2015 International Conference on Interactive Collaborative Learning (ICL), 838–843. https://doi.org/10.1109/ICL.2015.7318137 .

Martín-Gutiérrez J (2011) Proposal of methodology for learning of standard mechanical elements using augmented reality In: 2011 Frontiers in Education Conference (FIE), 1–6. https://doi.org/10.1109/FIE.2011.6142708 .

Martín-Gutiérrez J, Saorin JL, Contero M, Alcaniz M (2010) Ar_Dehaes: an educational toolkit based on augmented reality technology for learning engineering graphics In: 2010 10th IEEE International Conference on Advanced Learning Technologies, 133–137. https://doi.org/10.1109/ICALT.2010.45 .

Martín-Gutiérrez J, Navarro RE, González MA (2011) Mixed reality for development of spatial skills of first-year engineering students In: 2011 Frontiers in Education Conference (FIE), T2D–1–T2D–6. https://doi.org/10.1109/FIE.2011.6142707 .

Martínez AA, Benito JRL, González EA, Ajuria EB (2017) An experience of the application of augmented reality to learn english in infant education In: 2017 International Symposium on Computers in Education (SIIE), 1–6. https://doi.org/10.1109/SIIE.2017.8259645 .

Mayer RE (2009) Multimedia learning, 2 edn. Cambridge University Press, Cambridge.

Book   Google Scholar  

McGreal R (2009) Mobile devices and the future of free education. http://www.ou.nl/Docs/Campagnes/ICDE2009/Papers/Final_paper_252mcgreal.pdf . Accessed 07 Oct 2018.

Merriam SB (2009) Qualitative research: a guide to design and implementation. Jossey-Bass higher and adult education series. John Wiley & Sons. ISBN 9780470283547. https://books.google.at/books?id=tvFICrgcuSIC . Accessed 2 Oct 2018.

Moher D, Liberati A, Tetzlaff J, Altman DG (2010) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 8(5):336–341. ISSN 1743-9191. https://doi.org/10.1016/j.ijsu.2010.02.007 . http://www.sciencedirect.com/science/article/pii/S1743919110000403 .

Moura AMC (2010) Apropriação do Telemóvel como Ferramenta de Mediação em Mobile Learning: Estudos de Caso em Contexto Educativo. PhD thesis. Universidade do Minho. http://repositorium.sdum.uminho.pt/handle/1822/13183 . Accessed 2 Oct 2018.

Nakamura J, Csikszentmihalyi M (2014) The concept of flow. Springer Netherlands, Dordrecht. ISBN 978-94-017-9088-8. https://doi.org/10.1007/978-94-017-9088-8_16 .

Ontario Ministry of Education (2013) Inquiry-based learning. Technical report, Ontario Ministry of Education. http://www.edu.gov.on.ca/eng/literacynumeracy/inspire/research/CBS_InquiryBased.pdf .

Oxford RL (2001) Learning styles and strategies. In: Celce-Murcia M (ed)Teaching English as a Second or Foreign Language, 359–366.. Heinle Cengage Learning, Boston.

Ramírez H, Mendoza E, Mendoza M, González E (2015) Application of augmented reality in statistical process control, to increment the productivity in manufacture. Procedia Comput Sci 75:213–220. ISSN 1877-0509. https://doi.org/10.1016/j.procs.2015.12.240 . http://www.sciencedirect.com/science/article/pii/S1877050915037011 .

Roberto R, Lima JP, Teichrieb V (2016) ISSN 0097-8493. https://doi.org/10.1016/j.cag.2016.02.002 . http://www.sciencedirect.com/science/article/pii/S0097849316300115 . Comput Graph 56:20–30.

Salazar M, Gaviria J, Laorden C, Bringas PG (2013) Enhancing cybersecurity learning through an augmented reality-based serious game In: 2013 IEEE Global Engineering Education Conference (EDUCON), 602–607. https://doi.org/10.1109/EduCon.2013.6530167 .

Santos MEC, Chen A, Taketomi T, Yamamoto G, Miyazaki J, Kato H (2014) Augmented reality learning experiences: survey of prototype design and evaluation. IEEE Trans Learn Technol 7(1):38–56. ISSN 1939-1382. https://doi.org/10.1109/TLT.2013.37 .

Scriven M (1991) Beyond formative and summative evaluation In: Evaluation and Education: A Quarter Century.

Sommerauer P, Müller O (2014) Augmented reality in informal learning environments: a field experiment in a mathematics exhibition. Comput Educ 79:59–68. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2014.07.013 . http://www.sciencedirect.com/science/article/pii/S036013151400164X .

Swan JE, Gabbard JL (2009) Survey of user-based experimentation in augmented reality In: 1st International Conference on Virtual Reality, HCI International, 1–9. http://people.cs.vt.edu/jgabbard/publications/hcivr05.pdf . Accessed 2 Oct 2018.

Tarng W, Ou KL (2012) A study of campus butterfly ecology learning system based on augmented reality and mobile learning In: 2012 IEEE Seventh International Conference on Wireless, Mobile and Ubiquitous Technology in Education, 62–66. https://doi.org/10.1109/WMUTE.2012.17 .

Tarng W, Yu CS, Liou FL, Liou HH (2013) Development of a virtual butterfly ecological system based on augmented reality and mobile learning technologies In: 2013 9th International Wireless Communications and Mobile Computing Conference (IWCMC), 674–679. https://doi.org/10.1109/IWCMC.2013.6583638 .

Tobar-Muñoz H, Fabregat R, Baldiris S (2014) Using a videogame with augmented reality for an inclusive logical skills learning session In: 2014 International Symposium on Computers in Education (SIIE), 189–194. https://doi.org/10.1109/SIIE.2014.7017728 .

Tori R, Kirner C, Siscoutto RA (2006) Fundamentos e tecnologia de realidade virtual e aumentada. Editora SBC. ISBN 9788576690689. https://books.google.com.br/books?id=h1H7PgAACAAJ . Accessed 2 Oct 2018.

Tsai CH, Huang JY (2014) A mobile augmented reality based scaffolding platform for outdoor fieldtrip learning In: 2014 IIAI 3rd International Conference on Advanced Applied Informatics, 307–312. https://doi.org/10.1109/IIAI-AAI.2014.70 .

Wang YS, Chen CM, Hong CM, Tsai YN (2013) Interactive augmented reality game for enhancing library instruction in elementary schools In: 2013 IEEE 37th Annual Computer Software and Applications Conference Workshops, 391–396. https://doi.org/10.1109/COMPSACW.2013.128 .

Wang Y-H (2017) Exploring the effectiveness of integrating augmented reality-based materials to support writing activities. Comput Educ 113:162–176. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2017.04.013 . http://www.sciencedirect.com/science/article/pii/S0360131517300969 .

Wei L, Najdovski Z, Abdelrahman W, Nahavandi S, Weisinger H (2012) Augmented optometry training simulator with multi-point haptics In: 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2991–2997. https://doi.org/10.1109/ICSMC.2012.6378250 .

Wei X, Weng D, Liu Y, Wang Y (2015) Teaching based on augmented reality for a technical creative design course. Comput Educ 81:221–234. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2014.10.017 . http://www.sciencedirect.com/science/article/pii/S0360131514002371 .

Wu H-K, Lee SW-Y, Chang H-Y, Liang J-C (2013) Current status, opportunities and challenges of augmented reality in education. Comput Educ 62(0):41–49. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2012.10.024 . http://www.sciencedirect.com/science/article/pii/S0360131512002527 .

Yen J-C, Tsai C-H, Wu M (2013) Augmented reality in the higher education: students’ science concept learning and academic achievement in astronomy. Procedia-Soc Behav Sci 103:165–173. ISSN 1877-0428. https://doi.org/10.1016/j.sbspro.2013.10.322 . http://www.sciencedirect.com/science/article/pii/S1877042813037671 .

Yeo CT, Ungi T, U-Thainual P, Lasso A, McGraw RC, Fichtinger G (2011) The effect of augmented reality training on percutaneous needle placement in spinal facet joint injections. IEEE Trans Biomed Eng 58(7):2031–2037. ISSN 0018-9294. https://doi.org/10.1109/TBME.2011.2132131 .

Yin RK (2009) Case study research: design and methods. Applied Social Research Methods. SAGE Publications. ISBN 9781412960991. https://books.google.com.br/books?id=FzawIAdilHkC .

Zhang J, Sung Y-T, Hou H-T, Chang K-E (2014) The development and evaluation of an augmented reality-based armillary sphere for astronomical observation instruction. Comput Educ 73:178–188. ISSN 0360-1315. https://doi.org/10.1016/j.compedu.2014.01.003 . http://www.sciencedirect.com/science/article/pii/S036013151400013X . Accessed 2 Oct 2018.

Zhou F, Duh HB-L, Billinghurst M (2008) Trends in augmented reality tracking, interaction and display: a review of ten years of ISMAR In: Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, 193–202.. IEEE Computer Society, Cambridge.

Download references

Acknowledgements

The authors would like to thank Rafael Roberto for his valuable contributions.

The authors would like to thank Fundação de Amparo a Ciência e Tecnologia de Pernambuco (FACEPE) (processes IBPG-0605-1.03/15) for partially funding this research.

Availability of data and materials

The datasets supporting the conclusions of this article are available at https://papercatalog.000webhostapp.com/ .

Author information

Authors and affiliations.

Voxar Labs, Centro de Informática, Universidade Federal de Pernambuco, Recife, Brazil

Manoela M. O. da Silva, João Marcelo X. N. Teixeira & Veronica Teichrieb

Departamento de Eletrônica e Sistemas, Universidade Federal de Pernambuco, Recife, Brazil

João Marcelo X. N. Teixeira

EDUMATEC, Centro de Educação, Universidade Federal de Pernambuco, Recife, Brazil

Patrícia S. Cavalcante

You can also search for this author in PubMed   Google Scholar

Contributions

MMOS contributed to the conceiving and designing of the research protocol as well as acquiring, analyzing, and interpreting the data and drafting the manuscript. JMXNT contributed to the conceiving and designing the research protocol, analyzing and interpreting the data, and drafting and revising the manuscript. VT and PSC contributed to the conceiving and designing of the research protocol, revising the manuscript, and coordinating the research. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Manoela M. O. da Silva .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional information

Authors’ information.

MMOS is a Ph.D. candidate in Computer Science at the Federal University of Pernambuco and a researcher at Voxar Labs. Her research interests include interactive media, evaluation of educational tools, augmented reality for education and teaching and learning of mother and foreign languages.JMXNT is an assistant professor at Electronics and Systems Department of Federal University of Pernambuco and senior scientist at Voxar Labs. His research interests include 3D tracking, augmented reality, computer vision, computer graphics and embedded systems. VT is an associate professor at the Federal University of Pernambuco and head of the Voxar Labs research group. Her research interests include augmented reality, visualization, tracking and interaction. PSC is an associate professor at the Federal University of Pernambuco and head of the GENTE - New Technologies and Education research group. Her research interests include MOOCS and teachers’ long life formations, Learning Spaces and Learning Scenarios.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

da Silva, M., Teixeira, J., Cavalcante, P. et al. Perspectives on how to evaluate augmented reality technology tools for education: a systematic review. J Braz Comput Soc 25 , 3 (2019). https://doi.org/10.1186/s13173-019-0084-8

Download citation

Received : 11 April 2018

Accepted : 10 January 2019

Published : 04 February 2019

DOI : https://doi.org/10.1186/s13173-019-0084-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Augmented reality
• Educational systems

LIACS Thesis Repository

These webpages contain theses and reports by students affiliated with the various bachelor and master programmes offered at the Leiden Institute of Advanced Computer Science ( LIACS ), the computer science and artificial intelligence department of Leiden University . Note: this thesis repository might be incomplete for certain programmes.

PhD Computer Science

The following thesis was written by a student in the 2018 class of the PhD Computer Science programme at Leiden University.

Thesis details

Citation details

Schraffenberger, H.K. (Hanna), Arguably Augmented Reality, Relationships Between the Virtual and the Real, Thesis PhD Computer Science, LIACS, Leiden University, 2018.

Graduate Thesis Or Dissertation

Applications of augmented reality in the construction industry public deposited.

A big driver in the construction industry, like most others, is productivity. Naturally, everyone would like to get the maximum possible output, after providing the minimum possible input. With the growing age of technology, we are beginning to see more technology efforts aimed at being used as a tool for construction and construction related activities. A few examples of such technologies include the use of drones, GPS mapping, and virtual and augmented reality (VR & AR).This paper is an exploration of current and historical applications of augmented reality, specifically how these technologies have been used and are currently being used to display information. The aim is to learn more about some of the limitations and potential opportunities of using AR to display information. Furthermore, what effects does AR support have on user performance and behavior. We will begin this process by learning the definition and origin of AR and other related technologies. We will then look at how AR technologies have specifically been implemented in the construction and similar industries.

This paper also provides a tutorial on the development of an AR application. An AR application was developed for two reasons. One, to serve as proof of concept for our research team and second to explore how having access to an AR application affects a user’s performance and behavior. The goal was to develop an application that could both display details and information that could be toggled on and off using virtual buttons, and also to generate a 3D model that could be viewed from any angle, allowing the user to understand the complete geometry of the 3D model. This exploration serves as the first of many endeavors the University’s research teamhopes to undertake in the field of virtual and augmented reality.

• Gebremedhin, Mussie
• Civil, Environmental, and Architectural Engineering
• Goodrum, Paul M
• Hollowell, Matthew R
• Pfeffer, W.T.
• University of Colorado Boulder
• Civil engineering
• Virtual Reality
• Augmented Reality
• Masters Thesis
• In Copyright
• English [eng]

Relationships

Interactive Media Systems, TU Wien

Virtual and Augmented Reality

Contact: Hannes Kaufmann

We conduct basic and application oriented research in all areas related to virtual and augmented reality.

Available Now

In progress.

iNetTutor.com

Online Programming Lessons, Tutorials and Capstone Project guide

Augmented and Virtual Reality Capstone and Thesis Project Ideas

List of Augmented and Virtual Reality Project Ideas

• Virtual Reality Platform for Educational and Learning Purposes
• Virtual Reality Campus Tour Experience using Unity3D
• Development of Interactive Lesson Material for Biology using Virtual Reality
• Virtual Reality Project for Diagnostic Imaging
• Interactive Museum Tour using Virtual Reality Technology
• Application of Virtual Reality in Physics Related Experiments
• Virtual Reality Earthquake Drill
• Design and Implementation of Virtual Reality Systems for Driving Simulation
• Augmented Reality for Learning Human Body
• ClassAR: An Augmented Reality Classroom
• Interactive Hotel Tour through Virtual Reality
• The use of Virtual and Augmented Reality in the area of e-commerce
• ARonDGo: AR Mobile Application for e-commerce
• Virtual Reality Based Integrated Traffic Simulation Project
• Advance Navigation and Direction for Tourism using Augmented Reality
• Augmented 3D model for Jewelry Shop
• VRHome: a Virtual Reality Experience for Real Estate Industry
• VRStories: Virtual Reality Story Telling App
• Building Virtual and Augmented Reality Museum Tour Experience
• Augmented Reality Home Assistant

Augmented Reality vs Virtual Reality, AR vs VR

Virtual reality  takes you away from the real world and completely blocks your sight with another digital environment. It is used in architecture, tourism, rehabilitation, healthcare, sports, entertainment.

Augmented Reality  brings non-existent objects into the real world transforming the surroundings with overlay imagery. It is used in education, arts, marketing, military, media, business.

https://thinkmobiles.com/blog/ar-vs-vr/

Tools for VR development

• Unreal Engine
• Autodesk 3ds Max
• Autodesk Maya
• AppGameKit -VR (AGK)

SDK for developing AR applications

• Apple ARKit
• Google ARCore
• Pikkart AR SDK

You may visit our  facebook  page for more information, inquiries and comments.

Hire  our team to do the project.

Post navigation

• Mobile Based Emergency Assistance App with Admin Portal
• Mobile Based Voting Application for Android and IOS

Similar Articles

Forecast Planner in Enrollment System

BMI Calculator in C#

Water Refilling System in PHP and Bootstrap