simulation software literature review

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Sensors (Basel)

Systematic Literature Review of Realistic Simulators Applied in Educational Robotics Context

Caio camargo.

1 Instituto Politécnico de Bragança, 5300-253 Bragança, Portugal; [email protected] (C.C.); tp.bpi@sevlacnog (J.G.)

José Gonçalves

2 CeDRI—Research Centre in Digitalization and Intelligent Robotics, 5300-253 Bragança, Portugal

3 INESC TEC—Institute for Systems and Computer Engineering, 4200-465 Porto, Portugal; tp.pu.ef@ocap

Miguel Á. Conde

4 Robotics Group, Engineering School, University of León, Campus de Vegazana s/n, 24071 León, Spain; [email protected]

Francisco J. Rodríguez-Sedano

Paulo costa.

5 Universidade do Porto, 4200-465 Porto, Portugal

Francisco J. García-Peñalvo

6 GRIAL Research Group, Computer Science Department, University of Salamanca, 37008 Salamanca, Spain; se.lasu@aicragf

This paper presents a systematic literature review (SLR) about realistic simulators that can be applied in an educational robotics context. These simulators must include the simulation of actuators and sensors, the ability to simulate robots and their environment. During this systematic review of the literature, 559 articles were extracted from six different databases using the Population, Intervention, Comparison, Outcomes, Context (PICOC) method. After the selection process, 50 selected articles were included in this review. Several simulators were found and their features were also analyzed. As a result of this process, four realistic simulators were applied in the review’s referred context for two main reasons. The first reason is that these simulators have high fidelity in the robots’ visual modeling due to the 3D rendering engines and the second reason is because they apply physics engines, allowing the robot’s interaction with the environment.

1. Introduction

With the development of computers, simulation has become a powerful tool in the many areas in which it can support design, planning, analysis and decision-making in research and development [ 1 , 2 , 3 , 4 , 5 ].

Simulation is the process of designing a model of an actual or theoretical physical system, executing the model and analyzing the output. It helps to understand our reality and its complexity by building artificial objects and dynamically acting out roles. The simulation application enables learning about something in a very effective way and, by modifying environment rules, we can observe the results of the interactions. It is also an interdisciplinary field, applied in all research fields in society, from engineering and computer science to economics and social science, and at all different scientific study levels, even to manufacturers. Researchers and companies may build experimental systems using simulators even in the early development stages, testing complexity, reality and specificity. The simulation tests can be gradually increased to a level where these virtual systems can help to solve real challenges of the physical world, create new revolutionary products and push human imagination and creative boundaries—one of the main applications of simulation in the robotics field. By designing new products and investigating performance, simulation permits the study of structures, characteristics and a robotic system’s function no matter how complex it is. Although, as the system’s complexity increases, the need for simulation rises at the same level. Hence, the simulation tools can, for sure, improve design, development and robotic operating systems. Simulators utilizing a graphical user interface and visualization tools can provide us with a realistic way of visualizing the robotics system’s operation [ 1 ].

Robot simulation started to become feasible and got more attention when the computational power of personal computers increased over the years in a significant way. In almost every computer today, it is possible to run complex algorithms and many graphical calculations. With that, realistic simulations are also possible thanks to the game industry’s efforts to create realistic visualisation in computer games. The creation of virtual worlds requires considerable processing power to render graphical environments and physics calculations. Consequently, this effort developed software engines that provide high-quality physics simulations and rendering software in the robotics domain [ 2 ].

In this context, physics engines are software that allow computers to create physics phenomena that we experience in the real world, that is, rigid body dynamics, collision detection, soft body dynamics, fluid dynamics and other physical aspects, and apply them to 3D objects in games (the most usual application) and other 3D renderings, which affects how those objects interact in the digital world. Game developers and video effects artists use physics engines to create lifelike computer-generated environments for video games, movies and television. Some architects may use physics engines to create realistic 3D renderings for concept designs. Even if a 3D environment does not require real-life physics, a physics engine will allow the designer to customise physics to fit their needs [ 6 , 7 ].Without something like a physics engine telling many different 3D objects how to interact, programming an environment would be extremely time-consuming. Some environments may have hundreds of objects that all interact with each other in various ways. For example, an object in a bowl on a table is interacting with the bowl, the other objects in the bowl, the table and the ground the table sits on. As a game developer or video effects artist, a physics engine will be part of the suite of tools applied to create 3D environments. In many cases, physics engines are included in game engines, 3D modeling suites and 3D rendering tools. However, it may be offered as a standalone or as a plug-in to another software [ 8 , 9 ].

To qualify as a physics engine, a software must:

• Simulate a variety of physical systems (rigid body dynamics, soft body dynamics, fluid dynamics, etc.);
• Apply those systems to 3D objects and environments;
• Work in tandem with other software systems to create a cohesive experience.

The main objective of this work is to present a systematic literature review that allows us to understand whether there are any realistic simulators that are or can be applied in an educational robotics context, and to obtain scientific databases in order to analyze and compare the features of these kinds of simulators. The reason for exploring the educational context is because of the multiple advantages for pre-university students of robotics application [ 10 ], specially for developing STEAM related competences [ 11 ]. Still within the context of educational robotics, this research seeks to find, analyze and compare realistic simulators capable of simulating robots, sensors and actuators in general. In order to answer the research question of this work and fulfill the goal, this review becomes important for future applications and frameworks that can be developed using these simulation tools to be applied at all educational levels and, as a consequence, in teaching robotics and computer science topics.

The structure of this work is as follows: Section 2 describes all the methodology followed to execute the systematic literature review [ 12 ], the research question, the PICOC method and the search string equation that was applied to the databases. Section 3 presents a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram with all the papers obtained from the database searches of the previous section. Section 4 discusses and analyzes the results from the selected and relevant papers, after filtering by selection criteria. Finally, in Section 5 , conclusions and future work are proposed.

This paper was conducted by following the systematic literature review methodology presented by Kitchenham [ 13 , 14 , 15 ]. A systematic literature review is a means of evaluating and interpreting all available research, relevant to a particular research question, topic area, or phenomenon of interest. The SLR aims to present a fair evaluation of a research topic using a trustworthy, rigorous and auditable methodology. The guidelines for conducting an SLR are divided into three phases: planning the review, conducting the review and reporting the review [ 16 , 17 , 18 , 19 , 20 , 21 , 22 ].

Before starting the planning of the SLR, a preliminary search is needed on a database, such as Google Scholar, to verify if there is an SLR with the same theme of research. If there is an SLR with the same topic, there would not be any need to conduct a new one [ 23 , 24 ]. In the case of this systematic review of the literature, no results were found, covering the realistic simulators subject, therefore, the SLR can be carried out as new research.

2.1. Planning the Review

The first part is the review planning, consisting of the process of identification and definition of the review execution, ensuring that the review is traceable [ 25 ]. At the beginning, it is necessary to clearly specify the research question that it aims to investigate. For this work, taking into account the described context in the Introduction, the research questions (RQs) are:

• RQ1: In the context of educational robotics, are there any realistic simulators capable of simulating any robot prototype?
• RQ2: Are these simulators capable of simulating the robot’s sensors and/or actuators?
• RQ3: Is such simulation based on physics engines?

Once the research questions have been defined, the PICOC method proposed by Petticrew and Roberts [ 22 ] was followed to define the review scope.

• Population (P): Robotics Simulators;
• Intervention (I): Realistic Robotics Simulators;
• Comparison (C): Compare the already existing robotics simulators;
• Outcome (O): Understand the ways of simulate realistic robots, being able to simulate micro-controllers, sensors and actuators as well;
• Context (C): Educational Robotics.

2.2. Inclusion and Exclusion Criteria

With the PICOC established, the scope of the review has been set, accompanied by the research questions and selection criteria—inclusion (IC) and exclusion (EC)—are defined to select the relevant papers that answer the research questions. For a paper to be selected, it has to meet all the Inclusion Criteria, and if it meets any Exclusion Criteria, it will be excluded.

• IC1: The papers are written in English; (AND)
• IC2: The papers are reported in peer reviewed conferences or journals or technical reports; (AND)
• IC3: The papers that use any kind of simulator, OR simulate realistic robotics, OR simulate sensors OR Actuators.

The Exclusion Criteria are the opposite of the Inclusion Criteria.

• EC1: The papers are NOT written in English; (OR)
• EC2: The papers are NOT reported in peer reviewed conferences or journals or technical reports; (OR)
• EC3: The papers that do NOT use any kind of simulator, OR simulate realistic robotics, OR simulate sensors OR Actuators.

These selection criteria will determine whether, from reading the paper’s title and abstract, it will be included in the review or not, and whether it is useful to include relevant works in the review in terms of its scope.

2.3. Search Methodology

The methodology of an SLR differs from a search made randomly on the Internet in several aspects. One of the most relevant is the need to determine the data sources, which should be the most important databases in terms of the research context. The electronic databases used in this work were: ACM Digital Library, IEEE Digital Library, ISI Web of Science, ScienceDirect, Scopus and Springer Link.

These databases were selected for three main reasons:

• They are well-known databases in this research field;
• They are relevant databases in the research theme of this literature review;
• It is possible to use a search string as well as Boolean operators to improve the results of the search process.

Given this procedure, the next step is to define the search string equation for the different databases. It was built using relevant terms from the PICOC methodology and they were connected by Boolean “AND” and “OR” operators [ 26 , 27 ]. Moreover, the asterisk sign operator was used to include both the singular and plural of each term. Taking this into account, the search string equation is shown as follows:

(“educational robotics” OR “educative robotics” OR “robotics and education”) AND (“realistic simulators” OR “prototype” OR “prototyping”)

The search string equation is divided into two main parts. The first part contains three related concepts, which are: “Educational Robotics”, “Educative Robotics” or “Robotics and Education”. These concepts are inclusive and connected between each other, and they were retrieved from the Context from the PICOC methodology. The search string equation will be executed in the all electronic databases in order to gather all the published papers connected with those areas.

The second part of the search equation is related to the main objective of this work, the terms: “Realistic Simulators”, “Prototype” or “Prototyping”. The “Realistic Simulators” term has the role of finding in the electronic databases all the papers that in some way have used a realistic simulator. The two last terms “Prototype” or “Prototyping” are related and help to expand our search, because these words represent one of the main applications for simulators, that is, prototype simulation.

The following describes and shows the search strings equation applied to each database.

[[All: “educational robotics”] OR [All: “educative robotics”] OR [All: “robotics and education”]] AND [[All: “realistic simulators”] OR [All: “prototype”] OR [All: “prototyping”]]

• IEEE Digital Library: In the IEEE Digital Library ( http://ieeexplore.ieee.org , accessed on 8 June 2021), we used the simple search bar on the web site, pasting the search strings there.
• ISI Web of Science: In the ISI Web of Science ( http://www.isiknowledge.com , accessed on 8 June 2021) he query terms were posted in the basic search tab to obtain the papers.
• ScienceDirect: For ScienceDirect ( http://www.sciencedirect.com , accessed on 8 June 2021), the use of the website was very straight forward; the equation was pasted in the search field to obtain the results from the database.

ALL(“Educational Robotics” OR “Educative Robotics” OR “Robotics and Education”) AND ALL(“Realistic Simulators” OR “Prototype” OR “Prototyping”).

• Springer Link: For the Springer Link database ( http://link.springer.com , accessed on 8 June 2021), the query string was used in the simple search bar on the website.

2.4. Quality Criteria

After the first preliminary part of paper selection, described as the Inclusion and Exclusion criteria, a new set of questions was defined to check the work’s quality before including them in the final literature review.

Each question can be answered with a possible weight between three values: 4.0 (Yes, it answers the question fully), 2.0 (Yes, it answers the question partially) and 0.0 (No, it does not answer the question). These values are assigned to the papers by reading them fully. The quality assessment checklist is shown in Table 1 .

Quality Assessment Checklist.

Therefore, each paper can be assigned a maximum of 44.0 points based on the quality criteria. In Figure 1 , it can be observed the distribution of these quality data.

Distribution of Quality Data.

The median overall score (out of 44) of the 100 included studies was 30, and the mean overall score was 29.08. We, therefore, decided to set a cut-off score of 30 points. All those papers that exceeded this score were included in the final synthesis.

Data Extraction Form

When the quality assessment process of the papers was running out, a data extraction form was made with a set of questions to evaluate the simulators used during the reading of the works. These questions are shown in Table 2 .

Data Extraction Form.

For the first three questions (DQ1, DQ2 and DQ3), the answer is Boolean so it could be answered with “Yes” or “No”. The following three questions (DQ4, DQ5 and DQ6) should be answered with strings. DQ4 describes the method used in the paper for using the simulator; DQ5 describes whether the simulation was made under a mathematical–physical model; and DQ6 states the name of the simulator applied. The last two questions required selecting one possible option. DQ7 asks if the simulator is free to use, is fully paid or a mix of both and DQ8 asks if the simulator is an open-source platform or not. The result of this data extraction will be presented in the results section in a Table format, in which every mentioned simulator in the papers has a score equal to or above 30.0.

This section presents all the results obtained from the searches on the databases. The data compilation was divided into different phases according to the PRISMA flow diagram, shown in Figure 2 , which details the actions taken during the SLR process [ 28 , 29 ].

The Systematic Literature Review process. Adapted from [ 29 ].

This process was carried out following the methodology described in Section 2.3 . The search on the databases was performed (on 27 August 2020), carrying on with the paper selection process:

• First, the results retrieved from the initial search were 559 papers in total, distributed in 41 citations from the ACM Digital Library, 14 from the IEEE Digital Library, 22 papers from ISI Web of Science, 92 works from ScienceDirect, 204 citations from Scopus and 186 from Springer Link.
• After the search, all these references were uploaded and organized into the Parsifal ( https://parsif.al/ , accessed on 8 June 2021) (the main tool applied to conduct this SLR) and it detected 60 duplicated records that were consequently removed.
• As result, 499 works were retrieved from the previous step and they were analyzed through the reading of their titles, keywords and abstracts and applying the Inclusion and Exclusion Criteria. From this process, 434 articles were excluded because they did not meet the requirements, leading us to the next phase, with 65 papers.

Distribution of publication per year and source.

• After the evaluation of the papers’ quality, 15 papers that scored higher than or equal to 30 were selected, adding to them the 35 obtained from the reference checking of the previous phase. This resulted in a total of 50 selected works with which to compose the present review, which can be seen distributed by publication year and source in the Figure.

4. Discussion and Results

This section describes the results of the developed systematic literature review. How every simulator addresses the research questions ( RQ1, RQ2 and RQ3 ) made in the Section 2.1 is discussed, through the data extraction form presented in Table 2 . Taking this into account, further subsections point out the answers to the data questions by discussing three main issues: (1) the features of the simulators found in the review; (2) some interesting exceptions about papers included in the study; and (3) the features of the engines, on which the simulators are based. Finally, a subsection describes the Robot Operating System, which is an alternative method for writing the robots’ software in the simulators.

Table 3 presents all the papers selected, showing their quality scores, the name of the simulator used and its ability to simulate robots, and its sensors and actuators. Note that the columns tagged as d , e and f address the answers to Data Questions 1, 2 and 3, meaning whether the simulator is able to simulate robotics, sensors and actuators, respectively.

All the papers included in the SLR. In ( a ) All the selected studies about simulators. ( b ) The answer for Data Question 6 ( DQ6 ), in which the answer is the name of the simulator used in the refereed study. The ( c ) column is the paper’s score in the quality assessment. Lastly, ( d ), ( e ) and ( f ) columns concern the ability to simulate robotics, sensors and actuators, addressing Data Questions ( DQ1 ),( DQ2 ) and ( DQ3 ), respectively.

As shown in Table 3 , several simulators were found, and the most used from the selected papers were Sim-Two (22 times), Gazebo (7 times) and V-REP (4 times). Something to point out is that the papers with a score of 42 points or 44 points are those applied in the educational context that were able to simulate all the features inquired for the research questions, but with a few exceptions (such as Exception 1 and Exception 2 ) that are described in the next subsections.

Although the most used simulators were Sim-Two, Gazebo and V-REP, others were found through the reading process ( Table 3 ). The table presents a distribution of the simulators in this literature review.

As can be noticed, the most mentioned simulators found in the full-read process were: USARSim, Gazebo, Webots, Sim-Two, Stage/Player, V-REP, UberSim, MuRoSimF and the Microsoft Robotics Studio. To better understand the reason why the papers’ authors cite or use them, each simulator is investigated and described in the next subsection.

4.1. Simulators Features

In this subsection the features of the simulators found in the literature are described. Besides the papers studied during the research methodology, in this part it another search was made in Google Scholar in order to find papers that contain the simulators’ details and facts to add and support the information about the simulators presented in Table 4 . This is necessary because, as the analysis in Figure 3 shows, most of the papers included are from before 2016, and many of the features of the simulators may have changed through the years due to technology evolution. The simulator features can be found in Table 5 .

All the simulators mentioned throughout the full-read papers.

Simulators Features.

As can be observed, many simulators were found during the search. The main characteristic among some of them is that they are based on physics engines, which allows them to simulate the robot and the robot’s environment in a more realistic way. Another important feature that was noticed was that some of these simulators are defined with multiple simulation purposes (ie V-REP, Webots, Gazebo, SimTwo and others), which means they are able to simulate several types of robots, unlike others that only simulate one type of robot (i.e., ARGoS, RoSoS, UberSim, OpenHRP3, Khepera and others).

As a result of searching information about the simulators listed above, an Excel file was uploaded into a GitHub repository, a summarized Table showing only the simulator’s main features. The repository’s link can be found in Appendix A .

Finally, most of the simulators were developed in the 2000 s, and it can be observed that the most mentioned simulators in Table 5 still continue to have updates for current technologies since their launch. An exception to these simulators is the Microsoft Robotics Studio, which has been discontinued, as have UberSim and Khepera.

4.2. Exception Points

This subsection describes some simulators and papers that were considered exceptions, found during the research of this SLR. Although they do not completely fulfill the previously defined requisites, they represent relevant work that it is worth to mention.

• The first exception point to be discussed in the Table 3 is the Exception 1 found in the paper [ 61 ]. This paper has as its title: “Mathematical modelling, simulation and experimental verification of a Scara robot”, from Das, M. T., & Dülger, L. C. The authors developed a complete mathematical model of the Scara robot (Serpent 1 type robot), but the simulations carried out during the study were made using a numerical simulator such as MATLAB, and also they do not show how or which simulations were conducted. However, this paper could be replicated in another simulator such as the V-REP or Sim-Two, for example.
• The next point that stands out as an exception (marked as Exception 2 in Table 3 ), is the paper [ 75 ], Cervera, Enric, et al., “The robot programming network”. The authors present a system that allows the users to learn robotics topics in a virtual environment using a web-based laboratory with real robots or 2D/3D simulators. In this case, the system gathers tools that are fundamental for robotics learning such as learning the Robot Operating System use, including the possibility to try out in realistic and non-realistic simulators that are embedded into this web-based system.
• Another reference that is an exception is [ 105 ], where the authors design a simulator with a realistic visualization of the head of IRYS robot. Although the simulator, made using the Unreal Engine, is realistic enough in what concern the robot motion and appearance, it is just to simulate this robot. For that reason it was categorized as an exception.
• The last exception is the paper [ 106 ], in which the authors present an architecture for the management of a fleet of cleaning robots and, for this purpose, they design a simulator to evaluate its framework. The simulator has is called CleanSim and simulates map dirtiness; in this way, the authors can test their algorithm to improve the efficiency of the cleaning method. However, this is an exception for not being a realistic simulator based on a physics engine.

Future work could include the simulation shown in item 1 of this subsection, where the authors could replicate the modeling made in [ 61 ] with a realistic simulator. Another point to note is that, during the complete reading of the articles, some non-realistic simulators were found. However, they were discarded although they were used in educational contexts, as in the case of [ 105 , 106 ]; the main reason is because they were not based on a physics engine.

4.3. Physics Engines

Throughout the search, reading and analyzing each paper and simulator, one common point stands out, that they are built with physics engines. Table 6 shows the physics engines found and a classification in different columns depending on if they are a free or a propietary solution.

Physics Engines.

As noted in Table 6 , there are many available physics engines softwares; some are paid software and others free. In [ 7 ], an evaluation among five free physics engines is presented, and the author concludes that there is no general physics engine that performs best for any given task; each has its strengths and weaknesses. Taking this previous consideration into this work, simulators based on one or more physics engines will overlap the performance of those built with only one, and that is updated repeatedly.

4.4. Robot Operating System-ROS

Another common feature of the simulators found during the research was the Robot Operating System (ROS). ROS is a framework for writing robot software. It has several tools, libraries and conventions that aim to simplify the task of creating complex and robust robot behavior across a wide variety of robotics platforms. This framework emerged as an alternative way to create general-purpose robot software. ROS provides standard operating system services, such as hardware abstraction and low-level device control, the implementation of commonly used features, message-passing between processes, and package management. Sets of ROS processes in execution are represented in a graph architecture where processing occurs at nodes that can receive and send messages such as multiplex sensors, control, status, planning, actuator and others. Despite the importance of reactivity and low latency in robot control, ROS itself is not a real-time operating system. For this instance, ROS is an important, free and open-source tool in the robotics field, being widely used for makers, researchers and in the industry, and it is integrated into sundry simulators such as, Gazebo, Webots, MORSE, V-REP and others ( https://www.ros.org/ , accessed on 8 June 2021) [ 35 , 36 , 43 , 44 , 49 , 50 , 52 , 53 , 75 , 77 , 87 , 107 , 108 , 109 , 110 , 111 , 112 ].

5. Conclusions and Future Work

In this paper, a systematic literature review of realistic simulators applied in an educational context was conducted in order to evaluate whether there is any simulator capable of simulating a robot prototype using realistic world physics.

By performing this systematic review, questions were answered about the found papers, providing a current state-of-the-art and a view of this research field. During the review process, 559 papers were retrieved from six different electronic databases, from which 50 relevant papers were selected and included in this review, after applying the inclusion and exclusion criteria and the quality assessment. Table 7 shows how the selected papers address the research questions asked in Section 2.1 .

Selected papers that address the Research Questions.

Therefore, by reading, analyzing and gathering data from each relevant paper and simulator, some simulators have been shown to be promising tools to be used in the educational context for some reasons that we observed.

But first, coming back to answer the research questions (RQ1, RQ2 and RQ3): considering all the simulators presented in Table 3 , the frequency that was cited in the papers by the different authors, as shown in Table 4 , the simulators’ features studied in Section 4.1 and Table 5 , and finally, from the considerations made at the end of the previous paragraph and sections, it is possible to conclude that the simulators that can be easily applied in the educational context, are: Gazebo, Webots, SimTwo and V-REP.

The reasons for this are: firstly, the long time they have been available for use, that is, since their launch they continue to receive updates to keep up to date with the technology; The second characteristic observed was the number of platform and robots prototype variations (wheeled, legged, humanoids, drones and others) available to be used, or the possibility to add, configure and use a robot of your own in these simulators; in this way, allowing simulation in different environments, allowing a high level of abstraction with high fidelity in the simulation due to the use of physics engines. The third was the simulator’s ability to execute the simulations under one or more physics engines; this is an indicator of how realistic the simulation is. Another important feature of these simulators is that all of them have 3D vision of the robot and the environment, giving us the feeling of working with the real robot, without having it; Finally, is the capability of integrating with third party systems or protocols, for example, the integration of Robot Operating System (ROS), TCP/IP, MATLAB, LabView and others.

As future work, it could be interesting to produce a framework that provides a guideline for modelling an actuator, sensor or the entire robot in order to upload it into one of these simulators to test our own robots with different actuators or sensors, and to test them in different environments, such as a maze arena and line-following circuits.

Abbreviations

The following abbreviations are used in this manuscript:

Appendix A. Data Repository

https://github.com/caioorafael/Systematic-Literature-Review-of-Realistic-Simulators-to-be-applied-in-Educational-Context.git , accessed on 8 June 2021.

Author Contributions

All the authors have collaborate in the same way. All authors have read and agreed to the published version of the manuscript.

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Simulation Modelling in Healthcare: An Umbrella Review of Systematic Literature Reviews

Affiliations.

• 1 School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK. [email protected].
• 2 School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK.
• PMID: 28560492
• DOI: 10.1007/s40273-017-0523-3

Background: Numerous studies examine simulation modelling in healthcare. These studies present a bewildering array of simulation techniques and applications, making it challenging to characterise the literature.

Objective: The aim of this paper is to provide an overview of the level of activity of simulation modelling in healthcare and the key themes.

Methods: We performed an umbrella review of systematic literature reviews of simulation modelling in healthcare. Searches were conducted of academic databases (JSTOR, Scopus, PubMed, IEEE, SAGE, ACM, Wiley Online Library, ScienceDirect) and grey literature sources, enhanced by citation searches. The articles were included if they performed a systematic review of simulation modelling techniques in healthcare. After quality assessment of all included articles, data were extracted on numbers of studies included in each review, types of applications, techniques used for simulation modelling, data sources and simulation software.

Results: The search strategy yielded a total of 117 potential articles. Following sifting, 37 heterogeneous reviews were included. Most reviews achieved moderate quality rating on a modified AMSTAR (A Measurement Tool used to Assess systematic Reviews) checklist. All the review articles described the types of applications used for simulation modelling; 15 reviews described techniques used for simulation modelling; three reviews described data sources used for simulation modelling; and six reviews described software used for simulation modelling. The remaining reviews either did not report or did not provide enough detail for the data to be extracted.

Conclusion: Simulation modelling techniques have been used for a wide range of applications in healthcare, with a variety of software tools and data sources. The number of reviews published in recent years suggest an increased interest in simulation modelling in healthcare.

Publication types

• Computer Simulation*
• Delivery of Health Care / methods*
• Models, Theoretical*
• Research Design
• Systematic Reviews as Topic*

• Open access
• Published: 27 July 2018

A systematic literature review of simulation models for non-technical skill training in healthcare logistics

• Chen Zhang   ORCID: orcid.org/0000-0003-4057-4124 1 ,
• Thomas Grandits 2 ,
• Karin Pukk Härenstam 3 , 4 ,
• Jannicke Baalsrud Hauge 5 &
• Sebastiaan Meijer 2  

Advances in Simulation volume  3 , Article number:  15 ( 2018 ) Cite this article

14k Accesses

34 Citations

2 Altmetric

Metrics details

A Correction to this article was published on 24 April 2019

This article has been updated

Resource allocation in patient care relies heavily on individual judgements of healthcare professionals. Such professionals perform coordinating functions by managing the timing and execution of a multitude of care processes for multiple patients. Based on advances in simulation, new technologies that could be used for establishing realistic representations have been developed. These simulations can be used to facilitate understanding of various situations, coordination training and education in logistics, decision-making processes, and design aspects of the healthcare system. However, no study in the literature has synthesized the types of simulations models available for non-technical skills training and coordination of care.

A systematic literature review, following the PRISMA guidelines, was performed to identify simulation models that could be used for training individuals in operative logistical coordination that occurs on a daily basis. This article reviewed papers of simulation in healthcare logistics presented in the Web of Science Core Collections, ACM digital library, and JSTOR databases. We conducted a screening process to gather relevant papers as the knowledge foundation of our literature study. The screening process involved a query-based identification of papers and an assessment of relevance and quality.

Two hundred ninety-four papers met the inclusion criteria. The review showed that different types of simulation models can be used for constructing scenarios for addressing different types of problems, primarily for training and education sessions. The papers identified were classified according to their utilized paradigm and focus areas. (1) Discrete-event simulation in single-category and single-unit scenarios formed the most dominant approach to developing healthcare simulations and dominated all other categories by a large margin. (2) As we approached a systems perspective (cross-departmental and cross-institutional), discrete-event simulation became less popular and is complemented by system dynamics or hybrid modeling. (3) Agent-based simulations and participatory simulations have increased in absolute terms, but the share of these modeling techniques among all simulations in this field remains low.

Conclusions

An extensive study analyzing the literature on simulation in healthcare logistics indicates a growth in the number of examples demonstrating how simulation can be used in healthcare settings. Results show that the majority of studies create situations in which non-technical skills of managers, coordinators, and decision makers can be trained. However, more system-level and complex system-based approaches are limited and use methods other than discrete-event simulation.

Quality and safety in healthcare depend on the successful interaction between multiple teams, individuals, and support processes aimed at making the right resources, such as medications, medical equipment, information, and people, available at the right time [ 1 , 2 ]. Furthermore, in many healthcare settings, resource utilization must be prioritized such that the person most in need of a resource from a medical perspective will receive it. The cost of failure is high, both in terms of personal tragedies as well as the socio-economic burden of increased costs due to prolonged treatments or hospital stay [ 3 ].

Many of the everyday decisions regarding how resources will be used for patient care are made by individuals and networks of people performing coordinating functions, in the sense that they manage the timing and execution of many care processes of multiple patients. Their decisions often depend on judgements combining perspectives on the relevant medical conditions, the resources at hand, and the urgency of the situation; their decisions also depend on receiving information to help make sense of the situation as well as managing high stakes and competing goals [ 4 ].

Little is known about how these prioritizing and coordination skills are learned, how people performing them build their mental system models, what information and strategies they use, and which work practices are most successful. Most of the individuals performing coordination tasks are trained on the job in an unsystematic manner, and the knowledge remains, for the most part, tacit.

Simulation in healthcare is well known as a method for training individuals and teams in escalating situations surrounding individual patients [ 5 ]. To create meaningful simulations for training the non-technical skills used in coordination [ 6 ], there is a need to develop simulations of logistical challenges in a systematic manner as well as to describe and develop learning outcomes for the non-technical skills used in coordination. To support this development, it is important to know what types of logistical problems can be addressed by what types of simulations.

Logistics is one of many growing fields in healthcare management. This trend is driven by various societal impacts; population growth and an aging society have already put pressure on the operation of healthcare systems [ 7 , 8 ]. While healthcare logistics has been defined in various ways by researchers, in this paper, we define it as “operational handlings for the delivery of care, including its supportive services, from origination to recipient.” Focusing on the recipient of services, healthcare logistics could be patient-centric or material-centric. Patient-centric logistics relate to patient flows through the healthcare system. In this context, quality, safety, and efficiency of services for patients are keywords. Material-centric logistics address the positioning, storage, and circulation of goods and materials, such as blood and pharmaceutical products, within the hospital or the healthcare system.

Computer-based simulation plays an important role in the operational support of healthcare logistics. Generally, simulation can be useful in the design of complex social-technical systems [ 9 ]. As an innovative technology for adding analytical capacity, simulation can be used as an intermediate test in the (re)design of organizational rules and structures, workflow process management, performance, and avoidance of human errors [ 10 , 11 ]. More specifically, according to Jun et al. [ 12 ], simulation could provide benefits such as more effective redesign or innovation, deeper insights into barriers and incentives to adoption, and provision of an environment to “bench-test” final products prior to formal release. A change or improvement in real systems, however, might be expensive or dangerous, and balancing resource allocations is a central non-technical skill for healthcare professionals. Simulation adds value by providing a solution for training individuals to solve customized problems in a virtual, persuasive environment.

The application of discrete-event simulation in healthcare began to grow considerably at the end of the 1990s [ 12 ]; however, it remains unknown what type of simulations could be used to train, develop, and test non-technical aspects of coordination. Many types of simulation paradigms exist today. Discrete-event simulation, system dynamics, and agent-based simulation are the most utilized tools for modeling and analyzing systems according to the user’s interests and the specific task addressed. Discrete-event simulation is a tool for assessing the efficiency of delivery structures, forecasting changes in patient flow and examining resource efficiency in staffing [ 12 ]. System dynamics focus on the effect of structure on behavior [ 13 ]. Instead of addressing individual transactions, system dynamics is commonly used for higher level problems, such as strategic decision making, management controls, or policy changes [ 14 ]. Agent-based simulation is based on a “bottom-up” construction for the provision of emergent phenomena based on individual interactions of resource units [ 15 ].

Literature reviews have been conducted with explicit focus on the application of simulation in patient flow or material flow. However, previous literature reviews have been limited in at least one of the following aspects: (1) Reviews usually address simulation of healthcare logistics in a very narrow manner, analyzing a single key aspect such as low stakeholder engagement [ 16 ], a single simulation technique [ 17 ], or a single department; (2) most reviews have examined papers published before 2012.

This study is a continuation of the work by Dieckmann et al. [ 6 ], with a focus on the identification of available simulation models to provide meaningful training of non-technical skills in healthcare logistics. This is the perspective through which the literature was reviewed and understood. Given the large number of training simulations published, it is of interest to explore the diversity in this genre. The objective of this study is to provide a systematic literature review to answer the following research question:

What types of simulation models are currently available for training non-technical skills in handling logistical issues?

Search strategy

To answer the research question, the Web of Science Core Collection, the ACM Digital Library, and JSTOR were searched to retrieve articles focusing on simulation in healthcare logistics between 1998 and 2017. We utilized papers from these three databases because all of them rigorously select core journals and the keynote proceedings of conferences. The search terms were divided into the following two categories: patient-centric queries and material-centric queries. The papers were screened following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines.

The keywords were formulated by the individual reviewers to identify papers on relevant simulation techniques and investigated systems, as summarized in Table  1 . Keywords such as “healthcare,” “patient flow,” “pharma*,” “blood,” and “drug” specify the issues addressed. Keywords such as “simulation,” “system dynamics,” “simulator,” and “game” specify the research methods implemented.

Paper inclusion criteria

The criteria for inclusion in this review were that studies addressed the research question and strive to improve the performance of healthcare logistics. As the focus was logistical issues in healthcare management, publications regarding epidemiology, nutrition process improvement, and statistical analysis of health programs were not included. Abstracts, book reviews of limited length, and papers not granting access to full texts were also discarded. In addition to these general requirements, criteria for classification were implemented:

Application-oriented paper. The paper employs at least one simulation technique and presents a detailed scenario, or experiment, of a real-world healthcare system.

Subjective and methodological paper. The paper focuses on subjective and methodological perspectives on simulation techniques but might not report a use case (Fig.  1 ).

PRISMA flow diagram of assessment procedure and results: number of records included and excluded and reasons

The scope of the research and focus area were decided after the screening process. Simulation paradigms were classified based on statements from the authors; if no simulation technique was stated, the conceptualization framework was checked to determine its relevant category.

Following the retrieval of papers, discarding of duplicates, and review by the authors, the total number of essential publications was 294. The search identified 248 patient-centric and 46 material-centric papers. The patient-centric spectrum included 214 problem-solving papers, among which 114 utilized discrete-event simulation. For material-centric papers, discrete-event simulation was the dominant simulation paradigm as well. The number of publications for the past 5 years remained high, reinforcing our supposition that there is much knowledge to be gained from recent publications. The repository is available in the declaration.

For qualitative analysis, representative papers, listed in Table  2 , were identified. The papers featured statements of the relevant research questions or a description of the investigated system. We considered the number of publications utilizing different simulation techniques, scopes of research, and tools.

Following the screening process, we identified the question levels and the categories of addressed issues. The following question levels were derived: single department/unit, cross-department/unit, and cross-institutional. The category single department/unit included studies that model operation within a single department in an organization. The category cross-department/unit included studies that simulate multiple departments/units within the same organization. The category cross-institutional included the simulation modeling of interactions and flows between healthcare service providers in a large-scale network with widespread distribution regions. We identified the following categories of addressed issues: care pathway and appointment, staffing decision making, work procedures, specialized transport, facility design, healthcare systems, supply chain, inventory management, network distribution and dispatching, network configuration, procurement logistics, methodological contributions, and miscellaneous. The facility design was considered because architectural planning is a strategic decision that has a durable and profound effect on healthcare operation. The miscellaneous category included all research publications that we were not able to clearly classify into at least one of the abovementioned categories.

Logistical simulations—review

Discrete-event simulation

Discrete-event simulation has been applied to model and analyze all aspects of logistics management in healthcare. In particular, patient flow management and planning of staffing requirement are effective applications of this simulation technology. Our profiling of the literature is mostly in line with the findings of previous literature reviews; that is, discrete-event simulation is a useful tool with respect to improving patient flow, managing bed capacity, and scheduling and utilizing of resources [ 16 , 17 ]. DeRienzo et al. addressed the effect of nursing capacity by comparing different nursing sizes and demonstrated the applicability of supporting healthcare managers in handling operative tasks [ 18 ]. Devapriya et al. also developed a decision-supporting tool based on discrete-event simulation for the strategic planning of hospital bed capacity [ 19 ]. Bhattacharjee et al. analyzed appointment scheduling policies for patients to be treated by a medical scanning machine [ 20 ]. Vasilakis et al. developed a discrete-event simulation to study how long it took for patients to obtain their appointments from their referral [ 21 ]. Jørgensen et al. investigated internal blood logistics in hospitals and evaluated the effects of various management controls on the waiting times for accessing blood samples [ 22 ]. This simulation paradigm is most suitable for the realistic representation of processes in health services for analyzing “what-if” scenarios and assessing the performance of a logistical system.

System dynamics

System dynamics is used for organizational simulations. The paradigm is a mechanism-driven one for making decisions strategically for health services and resources from a global perspective. For instance, Rashwan et al. developed a system dynamics simulation to study bed blocking in Irish hospitals [ 23 ]. The focus was twofold: testing the policies for solving delayed discharges and envisaging the counterproductive and unintended consequences of these new policies [ 24 ]. Brailsford et al. simulated patient flow perspectives to identify system-wide bottlenecks [ 25 ]. Through the simulation, Lane et al. showed that the daily variation of used hospital bed capacity could not be balanced in the long run by simply increasing capacity; instead, optimal design of flows should be the core of the operation technology [ 26 ]. One paper investigated logistical outsourcing [ 27 ] and deployed system dynamics simulation with a sensitivity analysis for the evaluation and analysis of sustainability and economic performance. Content holders can use system dynamics simulation to envisage the complexity and identify opportunities and risks of the policies and management controls proposed.

Agent-based simulation

Agent-based simulation could be considered a means of soft computing in healthcare logistics. Agent-based simulation provides a gateway for understanding the behavior of distributed and connected service providers. The associated modeling and analysis are able to handle engineering system problems in complex networks. As an example, this paradigm was introduced to solve the coordination and collaboration difficulty of caregivers in a mental healthcare system [ 28 ]. The positive effect of coordination technology was confirmed by such modeling considerations. The local decision rules of caregivers were relevant for operative decision making with respect to successful provision of home help, a conclusion drawn from Marcon et al.’s work [ 29 ]. Bidding decisions made by distributers and suppliers in the pharmaceutical industry were studied in Jetly et al.’s work [ 30 ]. The performance of a multi-site network was simulated with pre-selected indicators, including the number of released products, degree of consolidation, and the return on assets. Multi-agent systems are not only effective for modeling flows between providers; they could also be applied in hospital environments. In Marin et al.’s work, patients, nurses, doctors, and the department as the manager are specified as agents with simple behavior rules [ 31 ]. With the support of multi-agent languages, the properties and relationships of actors could be simulated and validated for a specific social-technical environment.

Game and participatory simulation

Games and participatory simulation are life-like media that facilitate experimental learning. The use of such media enables the development of non-technical teamwork skills. For instance, Mustafee and Katsaliaki developed a pedagogical business game that simulated the blood supply chain [ 32 , 33 ]. The players were encouraged to propose different solutions, taking costs, time-efficiency, and stock levels of products into account. Focusing on quality of service in healthcare, a web-based organizational simulation was built and deployed for training referral and diagnostic skills [ 34 ]. The results showed that the usefulness of information on symptoms, diseases, and severity levels is associated with the perception of information sources. A board game provided valuable insight into the adoption of future technology in hospital logistical work [ 15 ]. Regarding the practical settings of two hospitals, the adoption of wearable technologies was reflected through role play. Such role playing could also be used to analyze the working environment in wards [ 35 ].

Hybrid modeling

Hybrid modeling is the efficient combination of various simulation modeling techniques described. Most hybrid modeling involves coupling discrete-event simulation and system dynamics. Two case studies, concerning infection control and regional social care system engineering, were simulated using hybrid models [ 36 ]. Zulkepli modeled an integrated ICU by combining system dynamics and discrete-event simulation [ 37 ]. The greatest advantage of hybrid modeling is the ability to integrate different simulation approaches and empirical data from different sources [ 38 ].

Analysis of trends

Discrete-event simulation has been the most prominent paradigm for modeling patient-centric logistics over the past decade. Between 2008 and 2017, as presented in Fig.  2 , more than half of the included papers used discrete-event simulation. However, the distribution of simulation techniques among different periods showed growing methodological diversity in recent years. The presence of system dynamics is observed in all periods, although the number of publications remained small. Agent-based simulation, games, and hybrid modeling were utilized only in the last decade. The specific simulation paradigm used was not stated in some studies, especially between 1998 and 2007, during which the majority of the methods used were classified as miscellaneous. Game-based methods were used in a few studies. Thus, interactive simulations are still quite new and rarely used.

Simulation paradigms for patient-centric logistics

The single category was the predominant level addressed in all periods, as shown in Fig.  3 . Between 2008 and 2012, the majority of studies addressed logistical issues at the single-unit level. The systems perspective was introduced between 2013 and 2017. Work addressing logistics issues at the cross-departmental and cross-institutional levels formed half of all research efforts. However, cross-institutional issues remained largely underexplored in the literature compared with other problems studied.

Research scope for patient-centric logistics

Discrete-event simulation was the most prominent simulation paradigm in material-centric approaches as well, as shown in Fig.  4 . The research theme started to develop in 2006, after which the number of publications and the diversity of the utilized paradigms increased. Despite the growth, six of 16 papers utilized discrete-event simulation, and only three papers utilized system dynamics, agent-based simulation, and/or hybrid modeling.

Simulation paradigms for material-centric logistics

Compared with patient-centric logistics, material-centric logistics was covered by a limited number of articles. Shah et al. had already stressed the underexplored potential of this area in 2004 [ 39 ]. We identified few publications on this subject during this period. The period 2013–2017 showed the largest output, but the volume was still not able to catch up with that of papers related to patient-centric logistics.

As shown in Fig.  5 , a growing number of papers have analyzed material handling between multiple units. However, simulation design for cross- and single-department logistics was lacking over the last 5 years, despite studies reporting on the need for improving hospital internal supply chains to reduce costs [ 40 ].

Research scope for material-centric logistics

Discussions

The literature review demonstrated that different simulation techniques could be utilized for different educational purposes, as summarized in Table  3 . Discrete-event simulation is suitable for operational problems, whereas strategic issues are better explored by system dynamics. Agent-based simulation stands out as a versatile tool because that agent method is object-oriented and flexible for describing the anatomy of complex systems formed by multiple actors. Healthcare logistics is a complex socio-technical system characterized by interconnected components and non-rational operation management. Agent-based simulation can explicitly model the interaction between system components, facilitating the understanding of overall performance under uncertainty and dynamics.

Games and participatory simulation are particularly useful for training at the tactical level because games help identify productive or counterproductive human actions. The strength of the agent-based method is the modeling and analysis of human behavior [ 33 , 41 ]. Healthcare logistics are largely characterized by non-rational operative decision making by medical personnel regarding needs of their patients. By modeling decisions at the agent level, it is possible to obtain insight into the reasoning process of decisions being made [ 42 ]. By involving these operational experts in participatory simulations, we can assess their perception of processes and healthcare system operations [ 43 ]. This effort delivers insight at another level of abstraction than technical, often discrete-event-based, simulations can provide.

Regarding training purposes, agent-based simulation and games are suitable for training negotiation and coordination in logistics, whereas discrete-event simulation and system dynamics can be utilized for reducing the uncertainty of decision-making processes by adding details to the model.

Participatory simulation is valuable for validating various simulators that model complex systems. The advantage of participatory simulation corresponds to the delivery structure of the investigated system, as healthcare logistics is carried out by collaborative efforts in which different professionals, knowledge, and skills work together.

Discrete-event simulation has the lowest requirement of technological preparation and is found to support all areas [ 26 , 44 , 45 , 46 ]. System dynamics and agent-based simulation might require formal methods and mathematics pertaining to system design, such as differential equations [ 47 ], decision theories [ 48 ], and game-based approaches [ 33 ].

Complex socio-technical systems, such as air traffic controls [ 49 ], routinely apply flow and logistical simulations. The studies examined in this review indicate a growing practice of implementing simulation in healthcare settings to create situations in which the non-technical skills of managers, coordinators, and decision makers can be trained and developed. Building on existing concepts from other industries [ 50 ], future applications might be hands-on training of teams using gaming and participatory simulation alongside empirical data to create situations for training tricky decision making, for strategic planning, and for exploring the effect of decisions on other parts of the system [ 51 ].

Our review yielded many examples of applications in healthcare, indicating that the issues of training of strategic or operational coordination and decision making in healthcare can all be addressed by simulation. The orthogonal simulation techniques are discrete-event simulation, system dynamics, agent-based simulation, game, and participatory simulation. For patient-centric logistics, discrete-event simulation in single-department/unit scenarios is the most dominant form of simulation, the maturity of which takes the lead over other categories by a large margin. As a systems perspective was applied, discrete-event simulation became less popular and was compensated for by system dynamics or hybrid modeling. The literature review showed that tools for logistical simulations vary in this field, with tools such as AnyLogic, Arena, NetLogo, and board games implemented most frequently. This is an extensive study analyzing the growth in the use of simulation in healthcare settings.

Lack of standardization

The number of miscellaneous simulations was significant, although discrete-event simulation, system dynamics, and agent-based simulation were well-established and well-standardized simulation techniques in many software packages. Most of the miscellaneous simulations were custom-made solutions. A focal point of these papers was implementing the modular design of protocols, revealing a lack of standards. Compared with processes in many other industries, healthcare processes are less standardized, and thus, composition of services varies. We believe that much effort could be saved by employing standardization in both healthcare processes and simulation formulism.

Lack of identification for material-centric logistics

In the domain of material-centric logistics, the focus is on inventory management and network distribution. A general lack of articles indicated limited research effort. One reason is that material-centric logistics is not an independent research stream yet—in many cases, the analysis of material-centric logistics is attached to a larger research project pertaining to physical distribution and logistical management.

Lack of complex system modeling and simulation

System dynamics, agent-based simulation, and hybrid modeling were underdeveloped for handling the complexity of social-technical systems. Digital transformation would change many aspects of the human-technology interaction in the provision of health services. A knowledge gap exists between the promise of future delivery of care that abolishes institutional boundaries and the current methods for testing and demonstrating functionalities. To bridge this gap, we require a better understanding of interconnected relationships between care providers and extensions to model individual-level requirements.

Limitations

The review has limitations. The search terms were formulated by the authors. As a result, the data search might not have been comprehensive. To eliminate the risk of omitting important contributions, the search terms combined keywords related to content and scientific methods, respectively. Second, although both journal and conference contributions were considered, the exclusion of abstracts and posters might lead to publication bias according to the Assessment of Multiple Systematic Reviews (AMSTAR) checklist for assessing the quality of systematic reviews. Because the aim of the review was to identify logistical simulations for training and education purposes, the exclusion is understandable for the short contributions that are not able to document the simulation models in a detailed manner. Therefore, publication bias is not prevalent in our literature review. The review only analyzed papers published since 1998. This approach was taken because the growth in the use of healthcare simulations started as Jun et al. surveyed the practical application of discrete-event simulation in healthcare [ 12 ], which was noted by Persson and Persson [ 52 ]. Therefore, a synthesis of the literature after 1998 should not distort the analysis.

General conclusion

The overview demonstrates that the simulation models available are mainly event-based, which is understandable. The strict regulations and rules associated with the medical field make process simulation particularly suited to handling issues in this area. These perceptions, together with the lack of literature on using agent-based simulation and participatory simulation, suggest a research direction involving the development of ontologies, architectures, and terminologies for their better acceptance in training and education of non-technical skills, with more problem-solving studies performed to demonstrate the corresponding benefits.

It is worth noting that the growth of digitalized healthcare occurs in parallel with the demographic change into an aging society. Currently, digital transformation, provision of homecare, and de-institutionalization are transferring practical applications into the decentralized paradigm. This effort requires coordination between caregivers and stakeholders. Agent-based simulation and participatory simulation can support comprehensive engineering to achieve quality and safety improvements. Therefore, agent-based simulation and participatory simulation are promising approaches for better handling healthcare logistics given current societal trends.

Change history

24 april 2019.

The original article [1] contains a previous iteration of author, Chen Zhang’s name.

Abbreviations

Miscellaneous

Hybrid simulation

Systems dynamics

Klein G, Feltovich PJ, Bradshaw JM, Woods DD. Common ground and coordination in joint activity. In: Rouse WB, Boff KR, editors. Organizational simulation. New Jersey: Wiley; 2005. p. 139–84.

Vincent C, Amalberti R. Safety strategies in hospitals. In: Safer healthcare. Cham: Springer International Publishing. p. 73–91.

Nemeth CP, Nunnally M, O’Connor MF, Brandwijk M, Kowalsky J, Cook RI. Regularly irregular: how groups reconcile cross-cutting agendas and demand in healthcare. Cogn Tech Work. 2007;9:139–48.

Article   Google Scholar  

Macrae C, Draycott T. Delivering high reliability in maternity care: in situ simulation as a source of organisational resilience. Saf Sci. 2016. https://doi.org/10.1016/j.ssci.2016.10.019 .

Crichton M, O’Connor P, Flin R. Safety at the sharp end: a guide to non-technical skills. Hampshire: Ashgate Publishing, Ltd; 2013.

Dieckmann P, Zeltner LG, Helsø A-M. “Hand-it-on”: an innovative simulation on the relation of non-technical skills to healthcare. Adv Simul. 2016;1:30.

Ulmanen P, Szebehely M. From the state to the family or to the market? Consequences of reduced residential eldercare in Sweden: from the state to the family. Int J Soc Welf. 2015;24:81–92.

Hagihara A, Hasegawa M, Hinohara Y, Abe T, Motoi M. The aging population and future demand for emergency ambulances in Japan. Intern Emerg Med. 2013;8:431–7.

Article   PubMed   Google Scholar  

Kriz WC. Creating effective learning environments and learning organizations through gaming simulation design. Simul Gaming. 2003;34:495–511.

Kriz WC. Types of gaming simulation applications. Simul Gaming. 2017;48:3–7.

Meijer S. The power of sponges: comparing high-tech and low-tech gaming for innovation. Simul Gaming. 2015;46:512–35.

Jun JB, Jacobson SH, Swisher JR. Application of discrete-event simulation in health care clinics: a survey. J Oper Res Soc. 1999;50:109–23.

Brailsford SC. System dynamics: what’s in it for healthcare simulation modelers. In: Proceedings of the winter simulation conference; 2008. p. 1478–83.

Google Scholar  

Koelling P, Schwandt MJ. Health systems: a dynamic system-benefits from system dynamics. In: Proceedings of the winter simulation conference; 2005. p. 1321–7.

Mustafee N, Katsaliaki K, Taylor SJE. Profiling literature in healthcare simulation. Simulation. 2010;86:543–8.

Schaefer JJ, Vanderbilt AA, Cason CL, Bauman EB, Glavin RJ, Lee FW, et al. Literature review: instructional design and pedagogy science in healthcare simulation. Simul Healthc. 2011;6:30–41.

Nestel D, Groom J, Eikeland-Husebø S, OʼDonnell JM. Simulation for learning and teaching procedural skills: the state of the science. Simul Healthc. 2011;6:10–3.

DeRienzo CM, Shaw RJ, Meanor P, Lada E, Ferranti J, Tanaka D. A discrete event simulation tool to support and predict hospital and clinic staffing. Health Informatics J. 2016;23:124–33.

Devapriya P, Strömblad CTB, Bailey MD, Frazier S, Bulger J, Kemberling ST, et al. StratBAM: a discrete-event simulation model to support strategic hospital bed capacity decisions. J Med Syst. 2015;39:130.

Bhattacharjee P, Ray PK. Simulation modelling and analysis of appointment system performance for multiple classes of patients in a hospital: a case study. Oper Res Health Care. 2016;8:71–84.

Vasilakis C, Sobolev BG, Kuramoto L, Levy AR. A simulation study of scheduling clinic appointments in surgical care: individual surgeon versus pooled lists. J Oper Res Soc. 2007;58:202–11.

Jørgensen P, Jacobsen P, Poulsen JH. Identifying the potential of changes to blood sample logistics using simulation. Scand J Clin Lab Invest. 2013;73:279–85.

Rashwan W, Ragab MA, Abo-Hamad W, Arisha A. Bed blockage in Irish hospitals: system dynamics methodology. In: Proceedings of the winter simulation conference; 2013. p. 3984–5.

Rashwan W, Abo-Hamad W, Arisha A. A system dynamics view of the acute bed blockage problem in the Irish healthcare system. Eur J Oper Res. 2015;247:276–93.

Brailsford SC, Lattimer VA, Tarnaras P, Turnbull JC. Emergency and on-demand health care: modelling a large complex system. J Oper Res Soc. 2004;55:34–42.

Lane DC, Monefeldt C, Rosenhead JV. Looking in the wrong place for healthcare improvements: a system dynamics study of an accident and emergency department. J Oper Res Soc. 2000;51:518–31.

Azzi A, Persona A, Sgarbossa F, Bonin M. Drug inventory management and distribution: outsourcing logistics to third-party providers. Strateg Outsourcing Int J. 2013;6:48–64.

Kalton A, Falconer E, Docherty J, Alevras D, Brann D, Johnson K. Multi-agent-based simulation of a complex ecosystem of mental health care. J Med Syst. 2016;40:39.

Marcon E, Chaabane S, Sallez Y, Bonte T, Trentesaux D. A multi-agent system based on reactive decision rules for solving the caregiver routing problem in home health care. Simul Model Pract Theory. 2017;74:134–51.

Jetly G, Rossetti CL, Handfield R. A multi-agent simulation of the pharmaceutical supply chain. J Simul. 2012;6:215–26.

Escudero-Marin P, Pidd M. Using ABMS to simulate emergency departments. In: Proceedings of the winter simulation conference; 2011. p. 1239–50.

Mustafee N, Katsaliaki K. The blood supply game. In: Proceedings of the winter simulation conference; 2010. p. 327–38.

Katsaliaki K, Mustafee N, Kumar S. A game-based approach towards facilitating decision making for perishable products: an example of blood supply chain. Expert Syst Appl. 2014;41:4043–59.

Basole RC, Bodner DA, Rouse WB. Healthcare management through organizational simulation. Decis Support Syst. 2013;55:552–63.

Mattarelli E, Fadel KJ, Weisband SP. Design of a role-playing game to study the trajectories of health care workers in an operating room. In: Proceedings of conference on human factors in computing systems; 2006. p. 1091–6.

Brailsford SC, Desai SM, Viana J. Towards the holy grail: combining system dynamics and discrete-event simulation in healthcare. In: Proceedings of the winter simulation conference; 2010. p. 2293–303.

Zulkepli J, Eldabi T, Mustafee N. Hybrid simulation for modelling large systems: an example of integrated care model. In: Proceedings of the winter simulation conference; 2012.

zen MB, Zabawa J. Modeling healthcare demand using a hybrid simulation approach. In: Proceedings of the winter simulation conference; 2016. p. 1535–46.

Shah N. Pharmaceutical supply chains: key issues and strategies for optimisation. Comput Chem Eng. 2004;28:929–41.

Article   CAS   Google Scholar  

Gaba DM. Adapting space science methods for describing and planning research in simulation in healthcare: science traceability and decadal surveys. Simul Healthc. 2012;7:27–31.

Liu Z, Rexachs D, Epelde F, Luque E. An agent-based model for quantitatively analyzing and predicting the complex behavior of emergency departments. J Comput Sci. 2017;21(Suppl C):11–23.

Kolb EMW, Schoening S, Peck J, Lee T. Reducing emergency department overcrowding: five patient buffer concepts in comparison. In: Proceedings of the winter simulation conference; 2008. p. 1516–25.

Kotiadis K, Tako AA, Vasilakis C. A participative and facilitative conceptual modelling framework for discrete event simulation studies in healthcare. J Oper Res Soc. 2014;65:197–213.

Demir E, Gunal MM, Southern D. Demand and capacity modelling for acute services using discrete event simulation. Health Syst. 2017;6:33–40.

Reynolds M, Vasilakis C, McLeod M, Barber N, Mounsey A, Newton S, et al. Using discrete event simulation to design a more efficient hospital pharmacy for outpatients. Health Care Manag Sci. 2011;14:223–36.

Pehlivan C, Augusto V, Xie X. Admission control in a pure loss healthcare network: MDP and DES approach. In: Proceedings of the winter simulation conference; 2013. p. 54–65.

Medeiros DJ, Swenson E, DeFlitch C. Improving patient flow in a hospital emergency department. In: Proceedings of the winter simulation conference; 2008. p. 1526–31.

Ferrin DM, Miller MJ, McBroom DL. Maximizing hospital finanacial impact and emergency department throughput with simulation. In: Proceedings of the winter simulation conference; 2007. p. 1566–73.

Dehlendorff C, Kulahci M, Andersen KK. Designing simulation experiments with controllable and uncontrollable factors for applications in healthcare. Appl Stat. 2011;60:31–49.

Duguay C, Chetouane F. Modeling and improving emergency department systems using discrete event simulation. Simulation. 2007;83:311–20.

Kittipittayakorn C, Ying K-C. Using the integration of discrete event and agent-based simulation to enhance outpatient service quality in an orthopedic department. J Healthc Eng. 2016;2016.

Persson MJ, Persson JA. Analysing management policies for operating room planning using simulation. Health Care Manag Sci. 2010;13:182–91.

Guo M, Wagner M, West C. Outpatient clinic scheduling—a simulation approach. In: Proceedings of the winter simulation conference; 2004. p. 1981–7.

Wijewickrama A, Takakuwa S. Simulation analysis of appointment scheduling in an outpatient department of internal medicine. In: Proceedings of the winter simulation conference; 2005. p. 2264–73.

Johansson B, Jain S, Montoya-Torres J, Hngan J. Integrating balanced scorecard and simulation modeling to improve emergency department performance in Irish hospitals. In: Proceedings of the winter simulation conference; 2010. p. 2340–51.

Lin Y, Zhang J. An ant colony optimization approach for efficient admission scheduling of elective inpatients. In: Proceedings of the annual conference on genetic and evolutionary computation; 2011. p. 15–6.

Helm JE, AhmadBeygi S, Van Oyen MP. Design and analysis of hospital admission control for operational effectiveness. Prod Oper Manag. 2011;20:359–74.

Yokouchi M, Aoki S, Sang H, Zhao R, Takakuwa S. Operations analysis and appointment scheduling for an outpatient chemotherapy department. In: Proceedings of the winter simulation conference; 2012.

Khurma N, Salamati F, Pasek ZJ. Simulation of patient discharge process and its improvement. In: Proceedings of the winter simulation conference; 2013. p. 2452–62.

Kooij R, Mes MRK, Hans EW. Simulation framework to analyze operating room release mechanisms. In: Proceedings of the winter simulation conference; 2014. p. 1144–55.

Huggins A, Claudio D, Waliullah M. A detailed simulation model of an infusion treatment center. In: Proceedings of the winter simulation conference; 2014. p. 1198–209.

Prodel M, Augusto V, Xie X. Hospitalization admission control of emergency patients using markovian decision processes and discrete event simulation. In: Proceedings of the winter simulation conference; 2014. p. 1433–44.

Giesen E, Ketter W, Zuidwijk R. Dynamic agent-based scheduling of treatments: evidence from the Dutch youth health care sector. In: Müller JP, Ketter W, Kaminka G, Wagner G, Bulling N, editors. Multiagent system technologies. Cham: Springer International Publishing; 2015. p. 173–99.

Chapter   Google Scholar  

Chen Y, Kuo YH, Balasubramanian H, Wen C. Using simulation to examine appointment overbooking schemes for a medical imaging center. In: Proceedings of the winter simulation conference; 2015. p. 1307–18.

Crowe S, Vasilakis C, Gallivan S, Bull C, Fenton M. Informing the management of pediatric heart transplant waiting lists: complementary use of simulation and analytical modeling. In: Proceedings of the winter simulation conference; 2015. p. 1654–65.

Chen P-S, Robielos RAC, Palaña PKVC, Valencia PLL, GY-H C. Scheduling patients’ appointments: allocation of healthcare service using simulation optimization. J Healthc Eng. 2015;6:259–80.

Kim S-H, Chan CW, Olivares M, Escobar G. ICU admission control: an empirical study of capacity allocation and its implication for patient outcomes. Manag Sci. 2015;61:19–38.

Khanna S, Boyle J, Good N, Bell A, Lind J. Analysing the emergency department patient journey: discovery of bottlenecks to emergency department patient flow. Emerg Med Australas. 2017;29:18–23.

Kim BBJ, Delbridge TR, Kendrick DB. Adjusting patients streaming initiated by a wait time threshold in emergency department for minimizing opportunity cost. Int J Health Care Qual Assur. 2017;30:516–27.

Ozcan YA, Tànfani E, Testi A. Improving the performance of surgery-based clinical pathways: a simulation-optimization approach. Health Care Manag Sci. 2017;20:1–15.

Bozzetto M, Rota S, Vigo V, Casucci F, Lomonte C, Morale W, et al. Clinical use of computational modeling for surgical planning of arteriovenous fistula for hemodialysis. BMC Med Inform Decis Mak. 2017;17:26.

Article   PubMed   PubMed Central   Google Scholar  

Vile JL, Allkins E, Frankish J, Garland S, Mizen P, Williams JE. Modelling patient flow in an emergency department to better understand demand management strategies. J Simul. 2017;11:115–27.

Demir E, Southern D. Enabling better management of patients: discrete event simulation combined with the STAR approach. J Oper Res Soc. 2017;68:577–90.

Chepenik L, Pinker E. The impact of increasing staff resources on patient flow in a psychiatric emergency service. Psychiatr Serv. 2017;68:470–5.

Saltzman R, Roeder T, Lambton J, Param L, Frost B, Fernandes R. The impact of a discharge holding area on the throughput of a pediatric unit. Serv Sci. 2017;9:121–35.

Toth DJA, Khader K, Slayton RB, Kallen AJ, Gundlapalli AV, O’Hagan JJ, et al. The potential for interventions in a long-term acute care hospital to reduce transmission of carbapenem-resistant enterobacteriaceae in affiliated healthcare facilities. Clin Infect Dis. 2017;65:581–7.

Steward D, Glass TF, Ferrand YB. Simulation-based design of ed operations with care streams to optimize care delivery and reduce length of stay in the emergency department. J Med Syst. 2017;41:162.

Tako AA, Kotiadis K. PartiSim: a multi-methodology framework to support facilitated simulation modelling in healthcare. Eur J Oper Res. 2015;244:555–64.

Monks T, Pearn K, Allen M. Simulation of stroke care systems. In: Proceedings of the winter simulation conference; 2015. p. 1391–402.

Djanatliev A, Meier F. Hospital processes within an integrated system view: a hybrid simulation approach. In: Proceedings of the winter simulation conference; 2016. p. 1364–75.

Alahäivälä T, Oinas-Kukkonen H. Understanding persuasion contexts in health gamification: a systematic analysis of gamified health behavior change support systems literature. Int J Med Inform. 2016;96:62–70.

Zhong X, Lee HK, Li J. From production systems to health care delivery systems: a retrospective look on similarities, difficulties and opportunities. Int J Prod Res. 2017;55:4212–27.

Lim ME, Worster A, Goeree R, Tarride J-É. Simulating an emergency department: the importance of modeling the interactions between physicians and delegates in a discrete event simulation. BMC Med Inform Decis Mak. 2013;13:59.

Vanderby S, Carter MW. An evaluation of the applicability of system dynamics to patient flow modelling. J Oper Res Soc. 2010;61:1572–81.

Chockalingam A, Jayakumar K, Lawley MA. A stochastic control approach to avoiding emergency department overcrowding. In: Proceedings of the winter simulation conference; 2010. p. 2399–411.

Djanatliev A, German R. Prospective healthcare decision-making by combined system dynamics, discrete-event and agent-based simulation. In: Proceedings of the winter simulation conference; 2013. p. 270–81.

Wolstenholme E. A patient flow perspective of U.K. health services: exploring the case for new “intermediate care” initiatives. Syst Dyn Rev. 1999;15:253–71.

Jansen FJA, Etman LFP, Rooda JE, Adan IJBF. Aggregate simulation modeling of an MRI department using effective process times. In: Proceedings of the winter simulation conference; 2012.

Zambrano F, Concha P, Ramis F, Neriz L, Bull M, Veloz P, et al. Improving patient access to a public hospital complex using agent simulation. In: Proceedings of the winter simulation conference; 2016. p. 1277–88.

Lamé G, Jouini O, Cardinal JS-L, Carvalho M, Tournigand C, Wolkenstein P. Patient-hospital communication: a platform to improve outpatient chemotherapy. In: Proceedings of the winter simulation conference; 2016. p. 2099–110.

April J, Better M, Glover F, Kelly J, Laguna M. Enhancing business process management with simulation optimization. In: Proceedings of the winter simulation conference; 2006. p. 642–9.

Bowers J, Ghattas M, Mould G. Exploring alternative routes to realising the benefits of simulation in healthcare. J Oper Res Soc. 2012;63:1457–66.

Chavis J, Cochran AL, Kocher KE, Washington VN, Zayas-Cabán G. A simulation model of patient flow through the emergency department to determine the impact of a short stay unit on hospital congestion. In: Proceedings of the winter simulation conference; 2016. p. 1982–93.

Patvivatsiri L. A simulation model for bioterrorism preparedness in an emergency room. In: Proceedings of the winter simulation conference; 2006. p. 501–8.

Comas M, Castells X, Hoffmeister L, Román R, Cots F, Mar J, et al. Discrete-event simulation applied to analysis of waiting lists: evaluation of a prioritization system for cataract surgery. Value Health. 2008;11:1203–13.

Kuhl ME. A simulation study of patient flow for day of surgery admission. In: Proceedings of the winter simulation conference; 2012.

Miller M, Ferrin D, Shahi N. Estimating patient surge impact on boarding time in several regional emergency departments. In: Proceedings of the winter simulation conference; 2009. p. 1906–15.

Roure M, Halley Q, Augusto V. Modelling and simulation of an outpatient surgery unit. In: Proceedings of the winter simulation conference; 2015. p. 1525–36.

Doğan NÖ, Unutulmaz O. Lean production in healthcare: a simulation-based value stream mapping in the physical therapy and rehabilitation department of a public hospital. Total Qual Manag Bus Excell. 2016;27:64–80.

Batarseh OG, Goldlust EJ, Day TE. SysML for conceptual modeling and simulation for analysis: a case example of a highly granular model of an emergency department. In: Proceedings of the winter simulations conference; 2013. p. 2398–409.

Schonherr O, Rose O. A general model description for discrete processes. In: Proceedings of the winter simulation conference; 2011. p. 2201–13.

Southard PB, Chandra C, Kumar S. RFID in healthcare: a six sigma DMAIC and simulation case study. Int J Health Care Qual Assur. 2012;25:291–321.

Santibáñez P, Chow VS, French J, Puterman ML, Tyldesley S. Reducing patient wait times and improving resource utilization at British Columbia Cancer Agency’s ambulatory care unit through simulation. Health Care Manag Sci. 2009;12:392–407.

Konrad RA, Lawley MA. Input modeling for hospital simulation models using electronic messages. In: Proceedings of the winter simulation conference; 2009. p. 134–47.

Hagtvedt R, Ferguson M, Griffin P, Jones GT, Keskinocak P. Cooperative strategies to reduce ambulance diversion. In: Proceedings of the winter simulation conference; 2009. p. 1861–74.

Maull RS, Smart PA, Harris A, AA-F K. An evaluation of ‘fast track’ in A&E: a discrete event simulation approach. Serv Ind J. 2009;29:923–41.

Roberts SD. Tutorial on the simulation of healthcare systems. In: Proceedings of the winter simulation conference; 2011. p. 1408–19.

Hosseini S, Jannat S. Discrete event simulation technique for evaluating performance of oncology department: a case study. In: Proceedings of the winter simulation conference; 2015. p. 1343–54.

Levin S, Garifullin M. Simulating wait time in healthcare: accounting for transition process variability using survival analyses. In: Proceedings of the winter simulation conference. 2015. p. 1252–60.

Levin S, Dittus R, Aronsky D, Weinger M, France D. Evaluating the effects of increasing surgical volume on emergency department patient access. Qual Saf Health Care. 2011;20:146–52.

CAS   Google Scholar  

Mahapatra S, Koelling CP, Patvivatsiri L, Fraticelli B, Eitel D, Grove L. Pairing emergency severity index5-level triage data with computer aided system design to improve emergency department access and throughput. In: Proceedings of the winter simulation conference; 2003. p. 1917–25.

McClean S, Barton M, Garg L, Fullerton K. A modeling framework that combines Markov models and discrete-event simulation for stroke patient care. ACM Trans Model Comput Simul. 2011;21:25.

Reindl S, Mönch L, Mönch M, Scheider A. Modeling and simulation of cataract surgery processes. In: Proceedings of the winter simulation conference; 2009. p. 1937–45.

Takakuwa S, Wijewickrama A. Optimizing staffing schedule in light of patient satisfaction for the whole outpatient hospital ward. In: Proceedings of the winter simulation conference; 2008. p. 1500–8.

Takakuwa S, Katagiri D. Modeling of patient flows in a large-scale outpatient hospital ward by making use of electronic medical records. In: Proceedings of the winter simulation conference; 2007. p. 1523–31.

Takakuwa S, Shiozaki H. Functional analysis for operating emergency department of a general hospital. In: Proceedings of the winter simulation conference; 2004. p. 2003–11.

Coelli FC, Ferreira RB, Almeida RMVR, Pereira WCA. Computer simulation and discrete-event models in the analysis of a mammography clinic patient flow. Comput Methods Prog Biomed. 2007;87:201–7.

Rohleder TR, Lewkonia P, Bischak DP, Duffy P, Hendijani R. Using simulation modeling to improve patient flow at an outpatient orthopedic clinic. Health Care Manag Sci. 2011;14:135–45.

Wang T, Guinet A, Belaidi A, Besombes B. Modelling and simulation of emergency services with ARIS and Arena. Case study: the emergency department of Saint Joseph and Saint Luc Hospital. Prod Plan Control. 2009;20:484–95.

Abo-Hamad W, Arisha A. Multi-criteria framework for emergency department in Irish hospital. In: Proceedings of the winter simulation conference; 2012.

Abo-Hamad W, Arisha A. Simulation-based framework to improve patient experience in an emergency department. Eur J Oper Res. 2013;224:154–66.

Wang Y, Lee LH, Chew EP, Lam SSW, Low SK, Ong MEH, et al. Multi-objective optimization for a hospital inpatient flow process via discrete event simulation. In: Proceedings of the winter simulation conference; 2015. p. 3622–31.

Zhao Y, Peng Q, Strome T, Weldon E, Zhang M, Chochinov A. Bottleneck detection for improvement of emergency department efficiency. Bus Process Manag J. 2015;21:564–85.

Rashwan W, Habib H, Arisha A, Courtney G, Kennelly S. An integrated approach of multi-objective optimization model for evaluating new supporting program in Irish hospitals. In: Proceedings of the winter simulation conference; 2016. p. 1904–15.

Rashwan W, Arisha A. Modeling behavior of nurses in clinical medical unit in university hospital: burnout implications. In: Proceedings of the winter simulation conference; 2015. p. 3880–91.

Bair AE, Song WT, Chen Y, Morris BA. The impact of inpatient boarding on emergency department crowding: a discrete-event simulation study. In: Proceedings of the spring computer simulation conference; 2009.

Bountourelis T, Eckman D, Luangkesorn L, Schaefer A, Nabors SG, Clermont G. Sensitivity analysis of an ICU simulation model. In: Proceedings of the winter simulation conference; 2012.

Hamrock E, Paige K, Parks J, Scheulen J, Levin S. Discrete event simulation for healthcare organizations: a tool for decision making. J Healthc Manag. 2013;58:110–24.

Centeno MA, Albacete C, Terzano DO, Carrillo M, Ogazon T. Project and process improvements in healthcare organizations: a simulation study of the radiology department at JMH. In: Proceedings of the winter simulation conference; 2000. p. 1978–84.

Ramis FJ, Palma JL, Baesler FF. The use of simulation for process improvement at an ambulatory surgery center. In: Proceedings of the winter simulation conference; 2001. p. 1401–4.

Pasin F, Jobin MH, Cordeau JF. An application of simulation to analyse resource sharing organisations among health-care organisations. Int J Oper Prod Manag. 2002;22:381–93.

Wiinamaki A, Dronzek R. Using simulation in the architectural concept phase of an emergency department design. In: Proceedings of the winter simulation conference; 2003. p. 1912–6.

Samaha S, Armel WS, Starks DW. The use of simulation to reduce the length of stay in an emergency department. In: Proceedings of the winter simulation conference; 2003. p. 1907–11.

Blasak RE, Starks DW, Armel WS, Hayduk MC. Healthcare process analysis: the use of simulation to evaluate hospital operations between the emergency department and a medical telemetry unit. In: Proceedings of the winter simulation conference; 2003. p. 1887–93.

Baesler FF, Jahnsen HE, DaCosta M. The use of simulation and design of experiments for estimating maximum capacity in an emergency room. In: Proceedings of the winter simulation conference; 2003. p. 1903–6.

Centeno MA, Giachetti R, Linn R, Ismail AM. A simulation-ilp based tool for scheduling ER staff. In: Proceedings of the winter simulation conference; 2003. p. 1930–8.

Schenk JR, Huang D, Zheng N, Allen TT. Multiple fidelity simulation optimization of hospital performance under high consequence event scenarios. In: Proceedings of the winter simulation conference; 2005. p. 936–42.

Spry CW, Lawley MA. Evaluating hospital pharmacy staffing and work scheduling using simulation. In: Proceedings of the winter simulation conference; 2005. p. 2256–63.

Hay AM, Valentin EC, Bijlsma RA. Modeling emergency care in hospitals: a paradox-the patient should not drive the process. In: Proceedings of the winter simulation conference; 2006. p. 439–45.

Wijewickrama AKA, Takakuwa S. Simulation analysis of an outpatient department of internal medicine in a university hospital. In: Proceedings of the winter simulation conference; 2006. p. 425–32.

Ballard SM, Kuhl ME. The use of simulation to determine maximum capacity in the surgical suite operating room. In: Proceedings of the winter simulation conference; 2006. p. 433–8.

Taaffe K, Johnson M, Steinmann D. Improving hospital evacuation planning using simulation. In: Proceedings of the winter simulation conference; 2006. p. 509–15.

VanBerkel PT, Blake JT. A comprehensive simulation for wait time reduction and capacity planning applied in general surgery. Health Care Manag Sci. 2007;10:373–85.

Rico F, Salari E, Centeno G. Emergency departments nurse allocation to face a pandemic influenza outbreak. In: Proceedings of the winter simulation conference; 2007. p. 1292–8.

Miller M, Ferrin D, Ashby M, Flynn T, Shahi N. Merging six emergency departments into one: a simulation approach. In: Proceedings of the winter simulation conference; 2007. p. 1574–8.

Patvivatsiri L, Montes EJ Jr, Xi O. Modeling bioterrorism preparedness with simulation in rural healthcare system. In: Proceedings of the winter simulation conference; 2007. p. 1155–60.

Paul JA, Hariharan G. Hospital capacity planning for efficient disaster mitigation during a bioterrorist attack. In: Proceedings of the winter simulation conference; 2007. p. 1139–47.

Song WT, Bair AE, Chih M. A simulation study on the impact of physician starting time in a physical examination service. In: Proceedings of the winter simulation conference; 2008. p. 1553–62.

Protil RM, Stroparo JR, Bichinho GL. Applying computer simulation to increase the surgical center occupation rate at a university hospital in Curitiba-Brazil. In: Proceedings of the winter simulation conference; 2008. p. 1609–16.

Nielsen AL, Hilwig H, Kissoon N, Teelucksingh S. Discrete event simulation as a tool in optimization of a professional complex adaptive system. Stud Health Technol Inform. 2008;136:247.

PubMed   Google Scholar  

Huschka TR, Denton BT, Narr BJ, Thompson AC. Using simulation in the implementation of an outpatient procedure center. In: Proceedings of the winter simulation conference; 2008. p. 1547–52.

Oddoye JP, Jones DF, Tamiz M, Schmidt P. Combining simulation and goal programming for healthcare planning in a medical assessment unit. Eur J Oper Res. 2009;193:250–61.

Raunak M, Osterweil L, Wise A, Clarke L, Henneman P. Simulating patient flow through an emergency department using process-driven discrete event simulation. In: Proceedings of the international conference on software engineering; 2009. p. 73–83.

Efe K, Raghavan V, Choubey S. Simulation modeling movable hospital assets managed with RFID sensors. In: Proceedings of the winter simulation conference; 2009. p. 2054–64.

Ferrand Y, Magazine M, Rao U. Comparing two operating-room-allocation policies for elective and emergency surgeries. In: Proceedings of the winter simulation conference; 2010. p. 2364–74.

Zeltyn S, Marmor YN, Mandelbaum A, Carmeli B, Greenshpan O, Mesika Y, et al. Simulation-based models of emergency departments: operational, tactical, and strategic staffing. ACM Trans Model Comput Simul. 2011;21:24.

Weng S-J, Tsai B-S, Wang L-M, Chang C-Y, Gotcher D. Using simulation and data envelopment analysis in optimal healthcare efficiency allocations. In: Proceedings of the winter simulation conference; 2011. p. 1295–305.

Geis GL, Pio B, Pendergrass TL, Moyer MR, Patterson MD. Simulation to assess the safety of new healthcare teams and new facilities. Simul Healthc. 2011;6:125.

Cabrera E, Luque E, Taboada M, Epelde F, Iglesias ML. ABMS optimization for emergency departments. In: Proceedings of the winter simulation conference; 2012.

Kuo Y-H, Leung JM, Graham CA. Simulation with data scarcity: developing a simulation model of a hospital emergency department. In: Proceedings of the winter simulation conference; 2012.

Mustafee N, Lyons T, Rees P, Davies L, Ramsey M, Williams MD. Planning of bed capacities in specialized and integrated care units: incorporating bed blockers in a simulation of surgical throughput. In: Proceedings of the winter simulation conference; 2012.

Rashwan W, Ragab M, Abo-Hamad W, Arisha A. Evaluating policy interventions for delayed discharge: a system dynamics approach. In: Proceedings of the winter simulation conference; 2013. p. 2463–74.

Shin SY, Balasubramanian H, Brun Y, Henneman PL, Osterweil LJ. Resource scheduling through resource-aware simulation of emergency departments. In: Proceedings of the international workshop on software engineering in health care; 2013. p. 64–70.

Amyot D. Real-time simulations to support operational decision making in healthcare. In: Proceedings of the summer computer simulation conference; 2013. p. 64–70.

Verma S, Gupta A. Improving services in outdoor patient departments by focusing on process parameters: a simulation approach. In: Proceedings of the winter simulation conference; 2013. p. 2250–61.

Yaylali E, Simmons J, Taheri J. Systems engineering methods for enhancing the value stream in public health preparedness: the role of Markov models, simulation, and optimization. Public Health Rep. 2014;129:145–53.

Pinto LR, Perpétuo IHO, de Campos FCC, Ribeiro YCNMB. Analysis of hospital bed capacity via queuing theory and simulation. In: Proceedings of the winter simulation conference; 2014. p. 1281–92.

Ozen A, Balasubramanian H, Samra P, Ehresman M, Li H, Fairman T, et al. The impact of hourly discharge rates and prioritization on timely access to inpatient beds. In: Proceedings of the winter simulation conference; 2014. p. 1210–20.

Kalbasi A, Krishnamurthy D, Rolia J, Singhal S. Simulation by example for complex systems. In: Proceedings of the winter simulation conference; 2014. p. 974–85.

Wurzer G, Lorenz WE. Causality in hospital simulation based on utilization chains. In: Proceedings of the symposium on simulation for architecture and urban design; 2014.

Aboueljinane L, Sahin E, Jemai Z, Marty J. A simulation study to improve the performance of an emergency medical service: application to the French Val-de-Marne department. Simul Model Pract Theory. 2014;47:46–59.

Ghanes K, Jouini O, Jemai Z, Wargon M, Hellmann R, Thomas V, et al. A comprehensive simulation modeling of an emergency department: a case study for simulation optimization of staffing levels. In: Proceedings of the winter simulation conference; 2014. p. 1421–32.

Zhou Z, Wang Y, Li L. Process mining based modeling and analysis of workflows in clinical care—a case study in a Chicago outpatient clinic. In: Proceedings of the IEEE international conference on networking, sensing and control; 2014. p. 590–5.

van Buuren M, Kommer GJ, van der Mei R, Bhulai S. A simulation model for emergency medical services call centers. In: Proceedings of the winter simulation conference; 2015. p. 844–55.

Franck T, Augusto V, Xie X, Gonthier R, Achour E. Performance evaluation of an integrated care for geriatric departments using discrete-event simulation. In: Proceedings of the winter simulation conference; 2015. p. 1331–42.

Ghanes K, Wargon M, Jouini O, Jemai Z, Diakogiannis A, Hellmann R, et al. Simulation-based optimization of staffing levels in an emergency department. Simulation. 2015;91:942–53.

Carmen R, Defraeye M, Van Nieuwenhuyse I. A decision support system for capacity planning in emergency departments. Int J Simul Model. 2015;14:299–312.

dos Santos M, Quintal RS, da PAC, Gomes CFS. Simulation of operation of an integrated information for emergency pre-hospital care in Rio de Janeiro municipality. Procedia Comput Sci. 2015;55:931–8.

Agor J, McKenzie K, Ozaltin O, Mayorga M, Parikh RS, Huddleston J. Simulation of triaging patients into an internal medicine department to validate the use of an optimization based workload score. In: Proceedings of the winter simulation conference; 2016. p. 3708–9.

Lee W, Shin K, Lee H-R, Shin H, Lee T. A structured approach for constructing high fidelity ED simulation. In: Proceedings of the winter simulation conference; 2016. p. 1950–60.

Pujowidianto NA, Lee LH, Pedrielli G, Chen C-H, Li H. Constrained optimizaton for hospital bed allocation via discrete event simulation with nested partitions. In: Proceedings of the winter simulation conference; 2016. p. 1916–25.

Augusto V, Xie X, Prodel M, Jouaneton B, Lamarsalle L. Evaluation of discovered clinical pathways using process mining and joint agent-based discrete-event simulation. In: Proceedings of the winter simulation conference; 2016. p. 2135–46.

Tiwari V, Sandberg WS. Perioperative bed capacity planning guided by theory of constraints. In: Proceedings of the winter simulation conference; 2016. p. 1894–903.

Thorwarth M, Rashwan W, Arisha A. An analytical representation of flexible resource allocation in hospitals. Flex Serv Manuf J. 2016;28:148–65.

Kuo Y-H, Rado O, Lupia B, Leung JMY, Graham CA. Improving the efficiency of a hospital emergency department: a simulation study with indirectly imputed service-time distributions. Flex Serv Manuf J. 2016;28:120–47.

Kadı D, Kuvvetli Y, Çolak S. Performance analysis of a university hospital blood laboratory via discrete event simulation. Simulation. 2016;92:473–84.

Cimellaro GP, Piqué M. Resilience of a hospital emergency department under seismic event. Adv Struct Eng. 2016;19:825–36.

Zhong X, Lee HK, Williams M, Kraft S, Sleeth J, Welnick R, et al. Staffing ratio analysis in primary care redesign: a simulation approach. In: Matta A, Sahin E, Li J, Guinet A, Vandaele NJ, editors. Health care systems engineering for scientists and practitioners. New York: Springer; 2016. p. 133–44.

Lambton J, Roeder T, Saltzman R, Param L, Fernandes R. Using simulation to model improvements in pediatric bed placement in an acute care hospital. Jona J Nurs Adm. 2017;47:88–93.

Bakker M, Tsui K-L. Dynamic resource allocation for efficient patient scheduling: a data-driven approach. J Syst Sci Syst Eng. 2017;26:448–62.

Weng S-J, Xu Y-Y, Gotcher D, Wang L-M. A pilot study of available bed forecasting system (ABFS) in the emergency healthcare network. In: Proceedings of the summer computer simulation conference; 2017.

Tànfani E, Testi A. Improving surgery department performance via simulation and optimization. In: Proceedings of the IEEE workshop on health care management; 2010.

Friemann F, Schönsleben P. Reducing global supply chain risk exposure of pharmaceutical companies by further incorporating warehouse capacity planning into the strategic supply chain planning process. J Pharm Innov. 2016;11:162–76.

Ramis FJ, Baesler F, Berho E, Neriz L, Sepulveda JA. A simulator to improve waiting times at a medical imaging center. In: Proceedings of the winter simulation conference; 2008. p. 1572–7.

Zeinali F, Mahootchi M, Sepehri MM. Resource planning in the emergency departments: a simulation-based metamodeling approach. Simul Model Pract Theory. 2015;53:123–38.

Wang J, Zhong X, Li J, Howard PK. Modeling and analysis of care delivery services within patient rooms: a system-theoretic approach. IEEE Trans Autom Sci Eng. 2014;11:379–93.

Luangkesorn KL, Bountourelis T, Schaefer A, Nabors S, Clermont G. The case against utilization: deceptive performance measures in inpatient care capacity models. In: Proceedings of the winter simulation conference; 2012.

Holm LB, Dahl FA. Simulating the influence of a 45% increase in patient volume on the emergency department of Akershus University Hospital. In: Proceedings of the winter simulation conference; 2010. p. 2455–61.

Miller MJ, Ferrin DM, Szymanski JM. Simulating six sigma improvement ideas for a hospital emergency department. In: Proceedings of the winter simulation conference; 2003. p. 1926–9.

Mackay M, Qin S, Clissold A, Hakendorf P, Ben-Tovim D, McDonnell G. Patient flow simulation modelling-an approach conducive to multi-disciplinary collaboration towards hospital capacity management. In: Proceedings of the international congress on modelling and simulation; 2013. p. 50–6.

Mes M, Bruens M. A generalized simulation model of an integrated emergency post. In: Proceedings of the winter simulation conference; 2012.

Khurma N, Bacioiu GM, Pasek ZJ. Simulation-based verification of lean improvement for emergency room process. In: Proceedings of the winter simulation conference; 2008. p. 1490–9.

Centeno MA, Lee MA, Lopez E, Fernandez HR, Carrillo M, Ogazon T. A simulation study of the labor and delivery rooms at JMH. In: Proceeding of the winter simulation conference; 2001. p. 1392–400.

Ramakrishnan S, Nagarkar K, DeGennaro M, Srihari K, Courtney AK, Emick F. A study of the CT scan area of a healthcare provider. In: Proceedings of the winter simulation conference; 2004. p. 2025–31.

Ruohonen T, Neittaanmäki P, Teittinen J. Simulation model for improving the operation of the emergency department of special health care. In: Proceedings of the winter simulation conference; 2006. p. 453–8.

Bountourelis T, Luangkesorn L, Schaefer A, Maillart L, Nabors SG, Clermont G. Development and validation of a large scale ICU simulation model with blocking. In: Proceedings of the winter simulation conference; 2011. p. 1143–53.

Al-Araidah O, Boran A, Wahsheh A. Reducing delay in healthcare delivery at outpatients clinics using discrete event simulation. Int J Simul Model. 2012;11:185–95.

Pasupathy R. Performance evaluation in a laboratory medicine unit. In: Proceedings of the winter simulation conference; 2013. p. 3972–3.

Taylor SJE, Abbott P, Young T, Grocott-Mason R. Student modeling & simulation projects in healthcare: experiences with Hillingdon Hospital. In: Proceedings of the winter simulation conference; 2014. p. 3650–61.

Oh C, Novotny AM, Carter PL, Ready RK, Campbell DD, Leckie MC. Use of a simulation-based decision support tool to improve emergency department throughput. Oper Res Health Care. 2016;9:29–39.

Eskandari H, Riyahifard M, Khosravi S, Geiger CD. Improving the emergency department performance using simulation and MCDM methods. In: Proceedings of the winter simulation conference; 2011. p. 1211–22.

Mielczarek B, Uziałko J. Using simulation to forecast the demand for hospital emergency services at the regional level. In: Proceedings of the winter simulation conference; 2012.

Perimal-Lewis L. Health intelligence: discovering the process model using process mining by constructing start-to-end patient journeys. In: Proceedings of the Australasian workshop on health informatics and knowledge management; 2014. p. 59–67.

Hosseini N, Taaffe K. Evaluation of optimal scheduling policy for accommodating elective and non-elective surgery via simulation. In: Proceedings of the winter simulation conference; 2014. p. 1377–86.

Esengul Tayfur TK. Allocation of resources for hospital evacuations via simulation. In: Proceedings of the winter simulation conference; 2007. p. 1148–54.

Ashby M, Miller M, Ferrin D, Flynn T. Simulating the patient move: transitioning to a replacement hospital. In: Proceedings of the winter simulation conference; 2007. p. 1562–5.

Pérez ES, Yepez LA, de la Mota IF. Simulation and optimization of the pre-hospital care system of the National University of Mexico using travelling salesman problem algorithms. In: Proceedings of the summer computer simulation conference; 2010. p. 364–70.

Güttinger D, Godehardt E, Zinnen A. Optimizing emergency supply for mass events. In: Proceedings of the 4th international ICST (Institute for Computer Sciences, social-informatics and telecommunications engineering) conference on simulation tools and techniques; 2011. p. 125–33.

Noreña D, Yamín L, Akhavan-Tabatabaei R, Ospina W. Using discrete event simulation to evaluate the logistics of medical attention during the relief operations in an earthquake in Bogota. In: Proceedings of the winter simulation conference; 2011. p. 2666–78.

Ullrich C, Van Utterbeeck F, Dejardin E, Debacker M, Dhondt E. Pre-hospital simulation model for medical disaster management. In: Proceedings of the winter simulation conference; 2013. p. 2432–3.

Noei S, Santana H, Sargolzaei A, Noei M. Reducing traffic congestion using geo-fence technology: application for emergency car. In: Proceedings of the international workshop on emerging multimedia applications and services for smart cities; 2014. p. 15–20.

Moon I-C, Bae JW, Lee J, Kim D, Lee H, Lee T, et al. EMSSIM: emergency medical service simulator with geographic and medical details. In: Proceedings of the winter simulation conference; 2015. p. 1272–84.

Gibson IW. An approach to hospital planning and design using discrete event simulation. In: Proceedings of the winter simulation conference; 2007. p. 1501–9.

Miller MJ, Ferrin DM, Shahi N, LaVecchia R. Allocating outpatient clinic services using simulation and linear programming. In: Proceedings of the winter simulation conference; 2008. p. 1637–44.

Ashby M, Ferrin D, Miller M, Shahi N. Discrete event simulation: optimizing patient flow and redesign in a replacement facility. In: Proceedings of the winter simulation conference; 2008. p. 1632–6.

Boucherie RJ, Hans EW, Hartmann T. Health care logistics and space: accounting for the physical build environment. In: Proceedings of the winter simulation conference; 2012.

Wurzer G. In-process agent simulation for early stages of hospital planning. Math Comput Model Dyn Syst. 2013;19:331–43.

Wurzer G, Lorenz WE, Rössler M, Hafner I, Popper N, Glock BMODYPLAN. Early-stage hospital simulation with emphasis on cross-clinical treatment chains. In: Proceedings of the symposium on simulation for architecture and urban design; 2015. p. 160–3.

Schaumann D, Pilosof NP, Date K, Kalay YE. A study of human behavior simulation in architectural design for healthcare facilities. Ann Dell Ist Super Sanita. 2016;52:24–32.

Cimellaro GP, Malavisi M, Mahin S. Using discrete event simulation models to evaluate resilience of an emergency department. J Earthq Eng. 2017;21:203–26.

Pulat PS, Kasap S, Splinter GL. Simulation study of an ideal primary care delivery system. Simulation. 2001;76:78–86.

Moeke D, van de Geer R, Koole G, Bekker R. On the performance of small-scale living facilities in nursing homes: a simulation approach. Oper Res Health Care. 2016;11:20–34.

Maroufkhani A, Lanzarone E, Castelnovo C, Di Mascolo M. A discrete event simulation model for the admission of patients to a home care rehabilitation service. In: Matta A, Sahin E, Li J, Guinet A, Vandaele NJ, editors. Health care systems engineering for scientists and practitioners. New York: Springer; 2016. p. 91–100.

Bhattacharjee P, Ray PK. Patient flow modelling and performance analysis of healthcare delivery processes in hospitals: a review and reflections. Comput Ind Eng. 2014;78:299–312.

Hu S, Heim JA. Developing domain-specific simulation objects for modeling clinical laboratory operations. In: Proceedings of the winter simulation conference; 2014. p. 1341–52.

Workman RW. Simulation of the drug development process: a case study from the pharmaceutical industry. In: Proceedings of the winter simulation conference; 2000. p. 1995–8.

Blau GE, Pekny JF, Varma VA, Bunch PR. Managing a portfolio of interdependent new product candidates in the pharmaceutical industry. J Prod Innov Manag. 2004;21:227–45.

Chen Y, Mockus L, Orcun S, Reklaitis GV. Simulation-optimization approach to clinical trial supply chain management with demand scenario forecast. Comput Chem Eng. 2012;40:82–96.

Perez-Escobedo JL, Azzaro-Pantel C, Pibouleau L. Multiobjective strategies for New Product Development in the pharmaceutical industry. Comput Chem Eng. 2012;37:278–96.

Chen Y, Pekny JF, Reklaitis GV. Integrated planning and optimization of clinical trial supply chain system with risk pooling. Ind Eng Chem Res. 2013;52:152–65.

El Afia A, Mezouar H. A global mapping of the Moroccan supply chain of hospital drugs, and a simulation of the dispensation process. In: Proceedings of the international conference on big data, cloud and applications; 2017.

Huyghe J, Nouwen M, Vanattenhoven J. Involving end-users in game based ideation: a case study in hospital logistics. In: Proceedings of the Nordic conference on human-computer interaction; 2016.

Best AM, Dixon CA, Kelton WD, Lindsell CJ, Ward MJ. Using discrete event computer simulation to improve patient flow in a Ghanaian acute care hospital. Am J Emerg Med. 2014;32:917–22.

Caputo AC, Pelagagge PM. Management criteria of automated order picking systems in high-rotation high-volume distribution centers. Ind Manag Data Syst. 2006;106:1359–83.

Katsaliaki K, Brailsford SC. Using simulation to improve the blood supply chain. J Oper Res Soc. 2007;58:219–27.

Vila-Parrish AR, Ivy JS, King RE. A simulation-based approach for inventory modeling of perishable pharmaceuticals. In: Proceedings of the winter simulation conference; 2008. p. 1532–8.

Jung JY, Blau G, Pekny JF, Reklaitis GV, Eversdyk D. Integrated safety stock management for multi-stage supply chains under production capacity constraints. Comput Chem Eng. 2008;32:2570–81.

Mustafee N, Taylor SJE, Katsaliaki K, Brailsford S. Facilitating the analysis of a UK national blood service supply chain using distributed simulation. Simulation. 2009;85:113–28.

Rossetti MD, Liu Y. Simulating SKU proliferation in a health care supply chain. In: Proceedings of the winter simulation conference; 2009. p. 2365–74.

Ren C, Wang W, He M, Shao B, Wang Q, Dong J. The use of simulation for Global Supply Network rationalization. In: Proceedings of IEEE international conference on service operations and logistics, and informatics. 2010. p. 276–81.

Babaï MZ, Syntetos AA, Dallery Y, Nikolopoulos K. Dynamic re-order point inventory control with lead-time uncertainty: analysis and empirical investigation. Int J Prod Res. 2009;47:2461–83.

Baesler F, Bastías A, Nemeth M, Martínez C. Blood centre inventory analysis using discrete simulation. In: Proceedings of the winter simulation conference; 2012.

Onggo BS. Elements of a hybrid simulation model: a case study of the blood supply chain in low-and middle-income countries. In: Proceedings of the winter simulation conference; 2014. p. 1597–607.

Gebicki M, Mooney E, Chen S-J, Mazur LM. Evaluation of hospital medication inventory policies. Health Care Manag Sci. 2014;17:215–29.

Duan Q, Liao TW. Optimization of blood supply chain with shortened shelf lives and ABO compatibility. Int J Prod Econ. 2014;153:113–29.

Baesler F, Nemeth M, Martínez C, Bastías A. Analysis of inventory strategies for blood components in a regional blood center using process simulation. Transfusion. 2014;54:323–30.

Wang K-M, Ma Z-J. Age-based policy for blood transshipment during blood shortage. Transp Res Part E Logist Transp Rev. 2015;80:166–83.

Leung N-HZ, Chen A, Yadav P, Gallien J. The impact of inventory management on stock-outs of essential drugs in Sub-Saharan Africa: secondary analysis of a field experiment in Zambia. PLoS One. 2016;11:e0156026.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Yurtkuran A, Emel E. Simulation based decision-making for hospital pharmacy management. In: Proceedings of the winter simulation conference; 2008. p. 1539–46.

Lee YM. Analyzing dispensing plan for emergency medical supplies in the event of bioterrorism. In: Proceedings of the winter simulation conference; 2008. p. 2600–8.

Lee YM, Ghosh S, Ettl M. Simulating distribution of emergency relief supplies for disaster response operations. In: Proceedings of the winter simulation conference; 2009. p. 2797–808.

Ozdamar L. Planning helicopter logistics in disaster relief. Spectrum. 2011;33:655–72.

Kulkarni NS, Niranjan S. Multi-echelon network optimization of pharmaceutical cold chains: a simulation study. In: Proceedings of the winter simulations conference; 2013. p. 3486–98.

Postacchini L, Ciarapica FE, Bevilacqua M, Mazzuto G, Paciarotti C. A way for reducing drug supply chain cost for a hospital district: a case study. J Ind Eng Manag. 2016;9:207–30.

Abdelkafi C, Beck BHL, David B, Druck C, Horoho M. Balancing risk and costs to optimize the clinical supply chain-a step beyond simulation. J Pharm Innov. 2009;4:96–106.

Alfonso E, Xie X, Augusto V, Garraud O. Modelling and simulation of blood collection systems: improvement of human resources allocation for better cost-effectiveness and reduction of candidate donor abandonment. Vox Sang. 2013;104:225–33.

Article   CAS   PubMed   Google Scholar  

Cho S-H, Jang H, Lee T, Turner J. Simultaneous location of trauma centers and helicopters for emergency medical service planning. Oper Res. 2014;62:751–71.

Liao H-C, Chang H-H. The optimal approach for parameter settings based on adjustable contracting capacity for the hospital supply chain logistics system. Expert Syst Appl. 2011;38:4790–7.

Akcay A, Martagan T. Stochastic simulation under input uncertainty for contract-manufacturer selection in pharmaceutical industry. In: Proceedings of the winter simulation conference; 2016. p. 2292–303.

Jacobs EA, Bickel WK. Modeling drug consumption in the clinic using simulation procedures: demand for heroin and cigarettes in opioid-dependent outpatients. Exp Clin Psychopharmacol. 1999;7:412–26.

Royston G, Dost A, Townshend J, Turner H. Using system dynamics to help develop and implement policies and programmes in health care in England. Syst Dyn Rev. 1999;15:293–313.

Mustafee N, Taylor SJE, Katsaliaki K, Brailsford S. Distributed simulation with COTS simulation packages: a case study in health care supply chain simulation. In: Proceedings of the winter simulation conference; 2006. p. 1136–42.

Muller J, Popke C, Urbat M, Zeier A, Plattner H. A simulation of the pharmaceutical supply chain to provide realistic test data. In: Proceedings of the international conference on advances in system simulation; 2009. p. 44–9.

Devi SP, Rao KS, Krishnaswamy S, Wang S. System dynamics model for simulation of the dynamics of corneal transplants. OPSEARCH. 2010;47:284–92.

Ng Adam TS, Sy C, Li J. A system dynamics model of Singapore healthcare affordability. In: Proceedings of the winter simulation conference; 2011. p. 1306–18.

Djanatliev A, German R, Kolominsky-Rabas P, Hofmann BM. Hybrid simulation with loosely coupled system dynamics and agent-based models for prospective health technology assessments. In: Proceedings of the winter simulation conference; 2012.

zen MB. Estimating future demand for hospital emergency services at the regional level. In: Proceedings of the winter simulation conference; 2013. p. 2386–97.

Elleuch H, Hachicha W, Chabchoub H. A combined approach for supply chain risk management: description and application to a real hospital pharmaceutical case study. J Risk Res. 2014;17:641–63.

Padilla JJ, Diallo SY, Kavak H, Sahin O, Sokolowski JA, Gore RJ. Semi-automated initialization of simulations: an application to healthcare. J Def Model Simul. 2015;13:171–82.

Taaffe K, Zinouri N, Kamath AG. Integrating simulation modeling and mobile technology to improve day-of-surgery patient care. In: Proceedings of the winter simulation conference; 2016. p. 2111–2.

Pitt M, Monks T, Crowe S, Vasilakis C. Systems modelling and simulation in health service design, delivery and decision making. BMJ Qual Saf. 2016;25:38–45.

Jahangirian M, Borsci S, Shah SGS, Taylor SJE. Causal factors of low stakeholder engagement: a survey of expert opinions in the context of healthcare simulation projects. Simulation. 2015;91:511–26.

Dangerfield BC. System dynamics applications to European health care issues. J Oper Res Soc. 1999;50:45–353.

Harrell CR, Price RN. Healthcare simulation modeling and optimization using MedModel. In: Proceedings of the winter simulation conference; 2000. p. 203–7.

Joustra P, van der Sluis E, van Dijk NM. To pool or not to pool in hospitals: a theoretical and practical comparison for a radiotherapy outpatient department. Ann Oper Res. 2010;178:77–89.

Morrison BP, Bird BC. Healthcare process analysis: a methodology for modeling front office and patient care processes in ambulatory health care. In: Proceedings of the winter simulation conference; 2003. p. 1882–6.

Baldwin LP, Eldabi T, Paul RJ. Simulation in healthcare management: a soft approach (MAPIU). Simul Model Pract Theory. 2004;12:541–57.

White KP. A survey of data resources for simulating patient flows in healthcare delivery systems. In: Proceedings of the winter simulation conference; 2005. p. 926–35.

Brailsford S. Overcoming the barriers to implementation of operations research simulation models in healthcare. Clin Investig Med Med Clin Exp. 2005;28:312–5.

Eldabi T, Paul RJ, Young T. Simulation modelling in healthcare: reviewing legacies and investigating futures. J Oper Res Soc. 2007;58:262–70.

Chahal K, Eldabi T. Applicability of hybrid simulation to different modes of governance in UK healthcare. In: Proceedings of the winter simulation conference; 2008. p. 1469–77.

Gupta D, Denton B. Appointment scheduling in health care: challenges and opportunities. IIE Trans. 2008;40:800–19.

Brailsford SC, Harper PR, Patel B, Pitt M. An analysis of the academic literature on simulation and modelling in health care. J Simul. 2009;3:130–40.

Katsaliaki K, Mustafee N, Taylor SJE, Brailsford S. Comparing conventional and distributed approaches to simulation in a complex supply-chain health system. J Oper Res Soc. 2009;60:43–51.

Forsythe L. Action research, simulation, team communication, and bringing the tacit into voice society for simulation in healthcare. Simul Healthc. 2009;4:143–8.

Seropian M, Lavey R. Design considerations for healthcare simulation facilities. Simul Healthc. 2010;5:338–45.

Katsaliaki K, Mustafee N. Applications of simulation within the healthcare context. J Oper Res Soc. 2011;62:1431–51.

LeBlanc VR, Manser T, Weinger MB, Musson D, Kutzin J, Howard SK. The study of factors affecting human and systems performance in healthcare using simulation. Simul Healthc. 2011;6:24–9.

Beliën J, Forcé H. Supply chain management of blood products: a literature review. Eur J Oper Res. 2012;217:1–16.

Jahangirian M, Naseer A, Stergioulas L, Young T, Eldabi T, Brailsford S, et al. Simulation in health-care: lessons from other sectors. Oper Res. 2012;12:45–55.

Hong TS, Shang PP, Arumugam M, Yusuff RM. Use of simulation to solve outpatient clinic problems: a review of the literature. South Afr J Ind Eng. 2013;24:27–42.

Holm LB, Dahl FA. Simulating the effect of physician triage in the emergency department of Akershus University Hospital. In: Proceedings of the winter simulation conference; 2009. p. 1896–905.

Günal MM, Pidd M. DGHPSim: supporting smart thinking to improve hospital performance. In: Proceedings of the winter simulation conference; 2008. p. 1484–9.

Yeon N, Lee T, Jang H. Outpatients appointment scheduling with multi-doctor sharing resources. In: Proceedings of the winter simulation conference; 2010. p. 3318–29.

Sugiyama T, Goryoda S, Inoue K, Sugiyama-Ihana N, Nishi N. Construction of a simulation model and evaluation of the effect of potential interventions on the incidence of diabetes and initiation of dialysis due to diabetic nephropathy in Japan. BMC Health Serv Res. 2017;17:833.

Download references

This work is supported by the Stockholm County Council, in the design of the study and collection, analysis, interpretation of data, and writing the manuscript.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

School of Engineering Sciences in Chemistry, Biotechnology and Health, Royal Institute of Technology, 2010, Röntgenvägen 1, 14152, Huddinge, Sweden

School of Engineering Sciences in Chemistry, Biotechnology and Health, Royal Institute of Technology, Hälsovägen 11, 14152, Huddinge, Sweden

Thomas Grandits & Sebastiaan Meijer

Pediatric Emergency Department, Karolinska University Hospital, Tomtebodavägen 18a, 17177, Stockholm, Sweden

Karin Pukk Härenstam

Department of Learning, Informatics, Management and Ethics, Karolinska Institute, Tomtebodavägen 18a, 17177, Stockholm, Sweden

School of Industrial Engineering and Management, Royal Institute of Technology, Mariekällgatan 3, 15144, Södertälje, Sweden

Jannicke Baalsrud Hauge

You can also search for this author in PubMed   Google Scholar

Contributions

CZ, TG, and SM designed the study and located the articles from the databases. CZ and TG performed the screening process. CZ, TG, SM, KH, and JH analyzed and interpreted the systematic review results. CZ and TG were responsible for writing the paper. KH, JH, and SM were the major contributors for revising the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chen Zhang .

Ethics declarations

Ethics approval and consent to participate.

Not applicable

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Zhang, C., Grandits, T., Härenstam, K.P. et al. A systematic literature review of simulation models for non-technical skill training in healthcare logistics. Adv Simul 3 , 15 (2018). https://doi.org/10.1186/s41077-018-0072-7

Download citation

Received : 06 March 2018

Accepted : 25 June 2018

Published : 27 July 2018

DOI : https://doi.org/10.1186/s41077-018-0072-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Logistical simulations
• Non-technical skills

Advances in Simulation

ISSN: 2059-0628

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An analysis of the academic literature on simulation and modelling in health care

• Published: 07 September 2009
• Volume 3 , pages 130–140, ( 2009 )

Cite this article

• S C Brailsford 1 ,
• P R Harper 2 ,
• B Patel 1 &

14k Accesses

173 Citations

Explore all metrics

This article describes a multi-dimensional approach to the classification of the research literature on simulation and modelling in health care. The aim of the study was to analyse the relative frequency of use of a range of operational research modelling approaches in health care, along with the specific domains of application and the level of implementation. Given the vast scale of the health care modelling literature, a novel review methodology was adopted, similar in concept to the approach of stratified sampling. The results provide new insights into the level of activity across many areas of application, highlighting important relationships and pointing to key areas of omission and neglect in the literature. In addition, the approach presented in this article provides a systematic and generic methodology that can be extended to other application domains as well as other types of information source in health-care modelling.

Similar content being viewed by others

Different uses of Bronfenbrenner’s ecological theory in public mental health research: what is their value for guiding public mental health policy and practice?

The role of artificial intelligence in healthcare: a structured literature review

The algorithm journey map: a tangible approach to implementing AI solutions in healthcare

Avoid common mistakes on your manuscript.

1. Introduction

Undertaking a review of modelling and simulation in health care is without doubt a Herculean task. This is a literature which, having carried out searches on consecutive days using the Web of Knowledge (WoK) bibliographic database (wok.mimas.ac.uk) and the search string ‘ ((healthcare or health care) and (modelling or modeling or simulation)) ’, was found to be expanding at the rate of about 30 articles a day. A search carried out on June 21, 2007 using the Ovid search engine ( www.ovid.com ) and the same search string resulted in 176 320 hits. It is hard to imagine how a single person, research group or academic department could begin to keep up with such a literature.

Nevertheless this is the task that the Research Into Global Healthcare Tool (RIGHT) project team set itself. RIGHT ( www.right.org.uk ) is a collaborative research venture between six UK universities, funded by the British Engineering and Physical Sciences Research Council. The aim of RIGHT is to assess the feasibility of applying to decision making in health care some of the best-practice modelling and simulation methods that are used to support decision making in other sectors, such as manufacturing industry and defence.

The first phase of the RIGHT project has involved eight extensive literature reviews, of which this is one. Nearly all of these involved massive literatures and therefore an innovative common methodology was devised and developed, in order to reduce the scope of the task to something achievable in the time available. The other review topics are: simulation and modelling in manufacturing industry, simulation and modelling in aerospace, simulation and modelling in defence, management methods (excluding simulation and modelling) in health care, management methods (excluding simulation and modelling) in manufacturing industry, stakeholder analysis and framework development.

The study was concerned only with modelling as understood by an operational researcher, namely a structured approach to understanding (and possibly, but not always, solving) a real-world problem through developing a simplified version of the real system. We were particularly, but not exclusively, interested in applications of simulation. This covered computer-based approaches such as discrete-event simulation, agent-based simulation and system dynamics, as well as role-playing or business-gaming simulations. In a medical context the word ‘model’ covers a wide range of meanings. Therefore, in order to avoid as far as possible clinical, biochemical, microbiological or pharmacological articles where the word model has a very technical and specialised meaning, we restricted the search criteria to the terms modelling and modeling .

The aim of undertaking this review, and indeed the other reviews in the RIGHT project, was not merely to produce an academic article. The overall aim of RIGHT is to produce a ‘toolkit’ of methods and an explanatory framework or user guide that will suggest, for a given type of health-care problem and a given set of available resources at the user's disposal, the most suitable method(s) to use. The RIGHT project is a feasibility study and the toolkit and user guide will be tested on a sample range of exemplar sites ( Naseer et al, 2009 ).

2. The ancestry of health-care modelling reviews

Several review articles over the years have been written on health-care modelling. These have tended to focus either on a specific modelling methodology, such as discrete event simulation (for example, Fone et al, 2003 ) or on the use of modelling for a specific health-care setting, such as clinics (for example, Jun et al, 1999 ).

One of the earliest review articles in this field, and possibly the first comprehensive review of health-care modelling, was by Fries (1976) , who compiled a list of 188 articles that the author grouped into 15 categories according to their area of application. These include forecasting demand, appointment systems, ambulance requirements and deployment, and health planning and programme evaluation. The articles were selected only if they used what Fries describes as ‘mathematical methods of modelling and solving decision problems that form the core of OR’. This bibliography was later supplemented with an additional 164 articles to make a total of 352 references ( Fries, 1979 ). The review covers more than a dozen mainstream OR journals of that time, up to 1979, as well as referencing chasing as appropriate. The author does not provide details of the full list of journals searched nor the selection criteria, but one imagines that the 352 articles cited represent a large proportion of the body of health-care modelling literature at that time.

Two separate review articles on computer simulation projects were published 1 year later in 1980 by Tunnicliffe Wilson (1980) . One article focused on applications to health-care population problems and the other on health-care facilities. Between them, they covered over 200 articles. A follow-up article by the same author ( Wilson, 1981 ) focussed on implementation issues as the author reported that from the 200 reviewed articles, only 16 studies reported recommendations that had been acted upon.

Towards the end of the 1980s, Smith-Daniels et al (1988) reviewed the literature pertaining to acquisition decisions, for example sizing of facilities and facility location, and allocation decisions, for example inpatient admissions scheduling. They covered a number of techniques including simulation, queueing theory, Markov chains and heuristics. A few years later, Klein et al (1993) presented a bibliography that included medical decision making and simulation modelling with a focus on planning models.

Jun et al (1999) surveyed articles on the application of discrete event simulation modelling to health-care clinics and systems of clinics, for example hospitals, outpatient clinics and emergency departments. A taxonomy of the literature is presented covering published articles over the previous 20 years and categorised under two main themes: patient flow and allocation of resources. No discussion is made on the adopted review methodology and thus it is not possible to ascertain how systematic and wide-ranging this review is.

More recently, Fone et al (2003) produced a systematic review of computer simulation modelling in population health and health-care delivery. It is fair to say that this article is the first in health-care modelling to adopt a rigorous systematic review process that is described in detail in the article, and involved the screening of some 2729 references that eventually were reduced to 182 using inclusion criteria. The focus is entirely on discrete event simulation and articles are grouped into four application areas. The authors comment that although the number of modelling articles has grown substantially in recent years, very few report on outcomes of implementation of models and so the value of modelling requires further research. It is of interest to note that nothing appears to have changed over the years since Tunnicliffe Wilson made similar observations in 1981 ( Wilson, 1981 ).

In summary, most of these previous reviews have focussed on simulation (and, in particular, discrete event simulation) or have included a broader range of OR and mathematical methodologies, but have focussed on specific application areas. Furthermore, most fail to describe the review process and presumably represent an exhaustive bibliography of articles from journals that happen to be searched by the review team. Certainly no systematic approach is reported, except that by Fone et al (2003) . This article therefore fills a gap in the review literature by producing an up-to-date review unrestricted by methodology or application, and based on a systematic heuristic sampling review process covering a vast body of literature.

3. Review methodology and the RIGHT Information Template (RIT)

Within the 2-year timescale of the RIGHT project (of which the first 4 months was assigned to the literature reviews), it was clearly impossible to carry out anything approaching an exhaustive systematic review of any of these massive literatures. Therefore a heuristic, sampling-based approach was adopted across all eight reviews, using a variety of methods to identify the key articles and the emerging issues. This methodology has more in common with stratified experimental sampling than the kind of exhaustive survey typically attempted in a conventional literature review, for example a Cochrane systematic review ( www.cochrane-handbook.org ), where the aim is to ensure that all articles that meet a clearly defined set of inclusion criteria are read. In order to achieve a consistent approach across all eight reviews, a common template called the RIT was developed. The RIT contains the fields as shown in Table 1 , which are recorded with fixed categories or free-text as appropriate. For this particular review, some of the fields of the standard RIT were modified slightly, as described at the end of this section. Some of the free-text fields in the RIT, in particular the ‘MethodName’ and the ‘FunctionalArea’ fields, were replaced by constrained lists in this study to facilitate quantitative analysis. The choice of specific methods was informed by the findings from the other RIGHT reviews.

This study was far broader in scope than any of the previous health-care modelling reviews described above. The source literature was mainstream academic journal publications, accessible through three of the most widely used academic electronic databases: JSTOR ( www.jstor.org ), SCOPUS ( www.scopus.com ) and ISI WoK ( www.wok.mimas.ac.uk ). More general web searches using Google showed that the ‘grey’ literature in this area is equally massive and is worthy of further study in its own right. It will be the subject of a follow-up article later in the project, as arguably some of the most widely implemented work appears in the grey literature rather than the academic literature. SCOPUS covers journal publications from more diverse sources than JSTOR, but concentrates on more recent publications. Despite innovations in medical technology, the nature of the problems arising in health-care management has remained remarkably similar over the years. The SCOPUS search was limited to articles published after 1990, but the JSTOR search was unrestricted by date in order to capture the significant but older publications.

The literature review methodology consisted of three stages ( Figure 1 ). In stage 1, a very broad set of search terms was used to produce an initial set of articles. The search string was ‘(health-care OR health care) AND (modelling OR modeling OR simulat * OR (system AND dynamic * ) OR markov * )’, appearing in the title, abstract or keywords. In stage 2, a subset of these articles was selected for abstract review by a combination of ‘relevance rating’ and reference chasing as described below. Overall, 16% of the stage 1 articles were selected for abstract review, although this varied from 10% to 25% across the three different literature sources. In stage 3, the abstracts of all the stage 2 articles were read and a further down-selection made for inclusion in the final data set. The criteria used at this stage were that the article described a genuine application of modelling or simulation to a health-care problem. Any duplicates were removed at this stage, although there were surprisingly few of these between JSTOR and SCOPUS. The suitability of the stage 3 articles was then verified by full-text reading. In all 22% of the stage 2 articles were judged suitable for final inclusion, resulting in a total data set of 342 articles (119 from SCOPUS, 163 from JSTOR and 60 from WoK). A summary of the search results for stage 1 and sample sizes for stages 2 and 3 is shown in Table 2 . The three stages required 3 months extensive work on searching, screening and recording required information, with approximately 20% of the time required for stage 1 and 40% each for stages 2 and 3.

Stages of the review methodology.

JSTOR and SCOPUS both provide ‘relevance ratings’ and these were used in stage 2 to rank the first 500 articles in both databases for abstract scanning. It was not possible to discover the exact algorithm used to determine this relevance rating, but it was clearly based on the frequency of occurrence of the search terms. Many articles were eliminated at this stage, for example book reviews, abstracts of conference presentations or cost-effectiveness analyses of drug treatments (given we wanted to exclude the clinical, biochemical, microbiological and pharmacological literature). However, WoK does not provide such a ranking and therefore the innovative bibliometric visualisation tool CiteSpace Chen (2004 , 2006 ) was used. Chen (2004 , 2006 ) has demonstrated various uses of citation information and network analysis for the scientific literature. In particular, co-citation networks are a useful analytical method for the task of reference chasing. A co-citation network is a graphical representation of the references cited by a given set of publications enabling key articles that are widely referenced by later authors (ie, highly connected nodes of the network) to be identified. Using Citespace, a network was constructed using the cited references and citation count details from the stage 1 WoK articles, in order to down-select a set of relevant publications for stage 2 review. This set consisted of the 491 most cited references by more than 2500 publications in WoK selected with co-citation network; hence, it was representative of outcome from usual reference-chasing by researchers.

4. Data collection and recording

For each of the 342 articles in the final data set, the following information was recorded in an Excel worksheet:

Funding source

Level of implementation

Functional area

Layer in the industry

Databases and processes for literature review

Year of publication

The ‘MethodName’ field from the standard RIT was expanded to allow up to three separate methods (primary, secondary and tertiary) to be recorded for each reviewed article, together with the software used, if stated. A two-level hierarchy was used to classify modelling and simulation methods in this review. For example, the high-level category ‘Simulation’ had eight sub-categories, including discrete-event, system dynamics, agent-based, distributed and Monte Carlo simulation. For each publication, a main method was assigned to the principal modelling approach employed in the study. A constrained set of method categories was used. Because many studies used more than one method, up to two subsidiary methods could be recorded. Thus, for instance, a study by Lehaney et al (1999) that used a Soft Systems (SSM) approach as a means to develop a discrete event simulation model would have two methods recorded, firstly its primary method, Simulation/Discrete Event Simulation, and secondly, Qualitative/SSM. Of the total 342 articles, 204 used only one method, 113 used two methods and 25 used three methods.

Similarly, data for the field ‘FunctionalArea’ were recorded at two levels. At the top level, four broad categories were used: stakeholder interest, clinical or organisational processes, patient care delivery planning and research/policy. A more detailed classification of health-care function used the following nine categories:

Finance, Policy, Governance, Regulation

Public Health, Community service planning

Patient behaviour/characteristics

Planning, System/resource utilisation

Quality management, Performance monitoring or review

Risk management, Forecasting

Workforce/Staff management

Up to three of these categories could be recorded: a primary function and up to two other subsidiary functions. Of the total 342 articles, 102 were classified in one function only, 149 were classified in two categories and 91 were classified in three categories.

‘Layer’ (in the industry) was recorded at three levels: policy or regulation; facilitation or commissioning; and operation. Data for the field ‘ImplementationLevel’ were rated according to a three-level scale of implementation (see the Results section for further details).

5. Validation and verification

Systematic review approaches such as the Cochrane review methodology have a formalised structure in which the search strategy is highly prescriptive, and the inclusion and exclusion criteria for articles are precisely defined. A systematic review is (in theory at least) repeatable by other researchers, with identical results apart from the possible inclusion of articles that were unpublished at the time of the original review. The methodology described in this article can be similarly validated and repeated. Moreover, by way of ‘reality check’, the final list of 342 references was scanned by all four authors (who have, between them, over 50 years experience in the field of health-care modelling) to verify that certain well known, important articles from the literature had in fact been found and that no misclassified articles had been included. The full data set of references will be made available on the RIGHT website.

6.1. Date of publication

The publication dates of the selected articles ranged from 1952–2007. However, the vast majority (82%) in our review was published after 1990. By decade, the percentages were: pre-1979: 7.0%; 1980–1989: 10.8%; 1990–1999: 36.3%; 2000–2007: 45.9%.

6.2. Country of origin

Each article was classified by the country in which the research study was carried out. When analysed by continent, the relative proportions were as follows: North America: 206 (60.2%); Europe: 84 (24.6%); Asia: 31 (9.1%); Africa: 10 (2.9%); Australasia: 6 (1.8%); South and Central America: 1 (0.3%). Four of the articles (1.2%) could not be classified by country. The vast majority of studies (85%), therefore, were undertaken in North America and Europe. Of the North American articles all but seven were conducted in the USA (the rest being Canada) and of the publications based in Europe, 55 of the 84 articles were from the UK. The preponderance of studies based in the US and UK is to a degree explained by the fact that the review was restricted to English language articles. However, it also almost certainly reflects the relatively high levels of health-care OR in these two countries.

6.3. Method

The majority of publications were found to fall into the categories of statistical analysis, statistical modelling, simulation and qualitative modelling. A smaller but significant number employ mathematical modelling, and very few fall into the remaining three categories, which are therefore aggregated and jointly classified as ‘Other’. Interestingly, where qualitative methods are used, they are very often a subsidiary method, whereas when mathematical modelling is used, it almost always forms the primary method. The primary method employed is shown in Figure 2 and Table 3 .

Analysis of method by primary and subsidiary classifications.

When the more detailed second level of the modelling methodology tree was examined, a very wide range of methods was found in each of the major categories. Table 4 shows those methods which were used at least three times.

Perhaps the most striking feature of this breakdown is the relatively low proportion of articles using these most common methods, with more than half of all articles having a primary method not shown in Table 4 . In all 53% of articles have a primary method that is not observed in more than two articles. This gives an indication of the very wide variety of methods evident in the review. It can be seen that the most common primary method is some form of regression analysis (23% of all articles).

Interestingly, some techniques such as process mapping and Monte Carlo simulation were more commonly used as subsidiary methods. Typically, for instance, Monte Carlo simulation was used for testing or as a method of probabilistic sensitivity analysis for another form of model (eg, a Markov model). Qualitative approaches often formed a precursor to the development of a quantitative model such as a discrete event simulation.

The distribution of methods by year of publication, Figure 3 , indicates that simulation and qualitative methods in particular are currently increasing in use. In contrast, other methods appear to have a similar uptake to the previous decade with mathematical modelling methods possibly in relative decline. The ‘Other’ category, for which the majority of articles are first observed post-2000, include spatial/GIS modelling, and system/software related methods such as UML (Unified Modeling Language) and IDEF (Integrated Definition Methods) for enterprise modelling and analysis.

Analysis of method by year.

6.4. Funding

The primary source of funding, where reported, is shown in Figure 4 . Funding sources were classified as commerce (such as consulting or commercial firms), academia (no formal Research Council grant/bursary provision), authorities (such as a Government organisation or agency), grants (funding bodies) or health services (such as direct funding from a hospital).

Primary source of funding by method.

Overall, 60% of published work reported no formal funding, with only 4% funded directly by health services organisations. Notably, commercial funding has been mainly restricted to simulation studies with no examples of qualitative or mathematical modelling. However, in contrast, simulation fares less well with formal grant funding compared with other methods.

6.5. Functional area

The breakdown of publications by the top-level classification was as follows: stakeholder interest: 38 (11%); clinical and organisational processes and setup: 79 (23%); patient care requirement profiles and delivery planning: 117 (34%); research and policy: 108 (32%). The distribution of articles within the more detailed categories described above is shown in Figure 5 , broken down by primary function and subsidiary functions. It demonstrates, for example, that planning and system/resource utilisation methods are predominant, and that unlike the other methods, quality management, performance monitoring and review methods are used more commonly as subsidiary methods.

Distribution of function by primary and subsidiary classifications.

Figure 6 shows the relationships between function and method. Two particular features are that simulation methods are dominant in planning and system/resource utilisation, whereas statistical methods are dominant in finance, policy, governance and regulation. Further, more detailed analysis showed clear tendencies for certain functions to be associated with each other. For example, quality management and performance review was often coupled with planning system and resource allocation.

Relationship between function and method.

6.6. Analysis by level of implementation

A key aspect of any study is the extent to which the model has actually been used in practice for its stated purpose. Each modelling study was rated according to a three-level scale of implementation: 1: Suggested (theoretically proposed by the authors); 2: Conceptualised (discussed with a client organisation); 3: Implemented (actually used in practice). The number of articles rated in each category was Suggested 171 (50%); Conceptualised 153 (44.7%); Implemented 18 (5.3%). Depressingly, these figures emphasise previous findings ( Wilson, 1981 ; Fone et al, 2003 ) that levels of implementation for models in health-care OR are very small indeed and have not improved since the 1980s. A large proportion of modelling studies do, however, reach a conceptualised stage whereby a coherent approach is specified in a practical context with a health-care organisation.

Figure 7 shows the levels of implementation for each method. Statistical analysis was always either conceptualised or suggested with no instances of implementation. The proportion of conceptualised to suggested was higher for qualitative and statistical modelling, compared with mathematical modelling and simulation methods.

Level of implementation by method.

7. Discussion

The aim of this review was to quantify and describe current levels of utilisation of modelling and simulation methods in health care, as reported in the mainstream academic literature. Ultimately, as part of the RIGHT project, this information will be used to develop an evidence-based ‘model selection toolkit’ to assist health-care professionals to choose an appropriate modelling approach to tackle a particular problem in a specific context. However, in a broader context, this study belongs to the family of health-care modelling reviews described earlier, and extends and develops some of this earlier work.

The findings on publication dates show a steadily increasing rate of publication in this field, with simulation and qualitative (soft) methods in particular rising in popularity. However, straightforward simulation studies are generally less successful in gaining Research Council funding, compared with other more complex methodologies. This is likely to be because Research Councils are generally looking for innovative experimental approaches, rather than standard methodologies with a proven track record. This is understandable, given that their role is to encourage new theoretical developments, but it does support the argument that the academic literature may not be the best place to look for practical applications of simulation. However, we have shown that simulation studies are generally more successful in attracting commercial funding.

In general, when considering funding sources, the academic literature shows a huge contrast with the ‘grey’ literature, as only 4% of studies were funded by a health service organisation. It is clear that the modelling work that is undoubtedly being undertaken within the health sector by business consultancies or by analysts employed within health-care organisations does not get written up for publication in academic journals.

The relationships between function and method suggest that certain business functions, such as finance, policy and regulation, are more likely to use statistical methods, arguably because these managers traditionally tend to have a more numerate background and are familiar with these approaches. On the other hand, simulation methods fare better in highly stochastic settings where the visual interface may be more important, such as resource utilisation and planning.

Overall levels of implementation are depressingly low and suggest that little has changed since previous review articles. Taylor et al (2009) report similar insights across the simulation modelling field, citing a lack of real-world involvement in published simulation modelling as a great, missed opportunity. Interestingly in our study, the implementation rates for statistical methods were particularly low. This may simply reflect the fact that such methods are very difficult for the lay person to understand, although they are of theoretical interest, so that a disproportionate number of statistical articles may get published in academic journals. This type of article often does not need a ‘client’ as such, as it may simply involve the application of some statistical method to secondary data derived from the literature. Conversely, qualitative approaches require a client as they cannot be used without interacting with human beings in some way. Therefore, it is less surprising that these methods report a comparatively high level of implementation.

When reflecting on the adopted methodology, a particular benefit of the approach was classifying the studies by more than one method that permitted co-associations to be explored. Likewise, there were benefits of allowing multiple functional areas that permitted examination of associations between functions and methods. A particular difficulty, however, was in constructing a viable taxonomy for all the methods. Eight categories were defined for this review but these are clearly open to debate. Having worked through the review process and resulting analyses, this is likely to assist in shaping future search criteria for bibliographic searches.

A key area of specific interest is the field of the so-called ‘grey literature’. It seems clear that many references to health-care modelling exist outside the domain of conventional journal publications. Commercial and promotional literature, website references and unpublished presentations, for instance, contain much of interest in this field. The challenge is to find a viable means of accessing and referencing these sources, which by definition are not recorded in conventional bibliographic databases. Despite this we believe that ‘grey literature’ may be centrally important in revealing lessons to be learned from the implementation of models in health care, an area that seems to be sorely absent in most of the research literature reviewed here.

In this review, the scope is limited to the specific area of OR type health-care modelling. This study begins to provide insights into the level of activity across many areas of application. It highlights important relationships and points to key areas of omission and neglect in the literature. Some of the key findings are summarised below:

The vast majority of studies were carried out in North America and Europe.

There is a preponderance of statistical approaches in the literature; however, simulation and qualitative modelling both currently appear to be enjoying a strong period of popularity, relative to earlier decades.

Qualitative methods are commonly used as a secondary method and often as a subsidiary to simulation.

Overall, an extraordinarily wide range of methods is revealed in the literature, and many of these methods are highly specific or bespoke to the project in question.

Simulation methods are prominent in planning and system/resource utilisation.

Statistical methods are prominent in the areas of finance, policy, governance and regulation.

In general there are few obvious strong associations and the data are highly varied.

Startlingly few studies report evidence of implementation, although a relatively large proportion do demonstrate a conceptualised model.

8. Conclusion

Clearly the literature in health-care simulation and modelling is vast and is expanding at a rapid rate. Moreover, this literature covers a very diverse range of applications with many interacting and overlapping areas. Added to this is the lack of standards and consistency in the use of key terms (for example, the use of the term ‘model’) between publications. The work of systematically reviewing and classifying the research literature in this area is therefore fraught with difficulties. Despite, and maybe because of this, there is great value in developing a viable taxonomy of the documented research. Such a framework provides a potential basis and structure for understanding the field as a whole.

The approach presented in this article provides a systematic and generic methodology that can be extended to review further areas of the literature as well as other types of information sources in health-care modelling and simulation. The field of Health Technology Assessment, for instance, is a fertile area of research in economic modeling, which could yield useful insights into the application of these techniques.

Given the multi-dimensional and relatively complex nature of this literature review, presentation is another important challenge. Here there is a role for visualisation tools (such as that presented by Citespace) to provide user-friendly, accessible means to graphically depict the key relationships in the analysis.

Chen C (2004). Searching for intellectual turning points: Progressive knowledge domain visualization. Proceedings of the National Academy of Sciences 101(1): pp 5303–5310.

Chen C (2006). CiteSpace II: Detecting and visualizing emerging trends and transient patterns in scientific literature. J Am Soc Inf Sci Tec 57 (3): 359–377.

Article   Google Scholar  

Fone D et al (2003). Systematic review of the use and value of computer simulation modelling in population health and health care delivery. J Public Health Med 25 (4): 325–335.

Fries BE (1976). Bibliography of operations research in health-care systems. Opns Res 24 (5): 801–814.

Fries BE (1979). Bibliography of operations research in health-care systems: An update. Opns Res 27 (2): 408–419.

Jun JB, Jacobson SH and Swisher JR (1999). Application of discrete-event simulation in health care clinics: A survey. J Opl Res Soc 50 (2): 109–123.

Klein RW, Dittus RS, Roberts SD and Wilson JR (1993). Simulation modeling and health-care decision making. Med Decis Making 13 (4): 347.

Lehaney B, Clarke SA and Paul RJ (1999). A case of an intervention in an outpatients department. J Opl Res Soc 50 (9): 877–891.

Naseer A, Eldabi T and Young T (2009). RIGHT: A toolkit for selecting healthcare modelling methods. J Simulation, forthcoming.

Smith-Daniels VL, Schweikhart SB and Smith-Daniels DE (1988). Capacity management in health-care services—Review and future-research directions. Decision Sci 19 (4): 889–919.

Taylor SJE et al (2009). Simulation modelling is 50! Do we need a reality check? J Opl Res Soc 60 (S1): S69–S82.

Wilson JCT (1980). A review of population health care problems tackled by computer simulation. Public Health 94: 174–182.

Wilson JCT (1981). Implementation of computer simulation projects in health care. J Opl Res Soc 32 (9): 825–832.

Download references

Acknowledgements

We thank the anonymous referees for their very helpful comments. We are also grateful for the support of the Engineering and Physical Sciences Research Council (EPSRC) under grant EP/E019900/1, and for the support and input from the other members of the RIGHT team.

Author information

Authors and affiliations.

University of Southampton, Southampton, UK,

S C Brailsford & B Patel

Cardiff University, Cardiff, UK,

Peninsula Medical School, Exeter, UK

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to P R Harper .

Rights and permissions

Reprints and permissions

About this article

Brailsford, S., Harper, P., Patel, B. et al. An analysis of the academic literature on simulation and modelling in health care. J Simulation 3 , 130–140 (2009). https://doi.org/10.1057/jos.2009.10

Download citation

Received : 01 January 2009

Accepted : 03 June 2009

Published : 07 September 2009

Issue Date : 01 September 2009

DOI : https://doi.org/10.1057/jos.2009.10

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• health care modelling
• literature review
• methodology
• Find a journal
• Publish with us
• Track your research

• Review article
• Open access
• Published: 10 July 2017

The effect of games and simulations on higher education: a systematic literature review

• Dimitrios Vlachopoulos 1 &
• Agoritsa Makri 2  

International Journal of Educational Technology in Higher Education volume  14 , Article number:  22 ( 2017 ) Cite this article

402k Accesses

291 Citations

41 Altmetric

Metrics details

The focus of higher education institutions is the preparation of future professionals. To achieve this aim, innovative teaching methods are often deployed, including games and simulations, which form the subject of this paper. As the field of digital games and simulations is ever maturing, this paper attempts to systematically review the literature relevant to games and simulation pedagogy in higher education. Two researchers collaborate to apply a qualitative method, coding and synthesizing the results using multiple criteria. The main objective is to study the impact of games and simulations with regard to achieving specific learning objectives. On balance, results indicate that games and/or simulations have a positive impact on learning goals. The researchers identify three learning outcomes when integrating games into the learning process: cognitive, behavioural, and affective. As a final step, the authors consolidate evidence for the benefit of academics and practitioners in higher education interested in the efficient use of games and simulations for pedagogical purposes. Such evidence also provides potential options and pathways for future research.

Introduction

As rapidly evolving technological applications, games and simulations are already widely integrated in the traditional educational process. They are deployed extensively in the field of education, with an existing body of work examining the relation between games and education (Yang, Chen, & Jeng, 2010 ; Chiang, Lin, Cheng, & Liu, 2011 ). In recent years, digital or web-based games have increasingly supported learning. In the context of online education, this research area attracts a significant amount of interest from the scientific and educational community, for example tutors, students and game designers. With the growing expansion of technology, instructors and those who create educational policy are interested in introducing innovative technological tools, such as video games, virtual worlds, and Massive Multi-Player Online Games (MMPOGs) (Buckless, 2014 ; Gómez, 2014 ).

Games and simulations show mixed effects across a number of sectors, such as student performance, engagement, and learning motivation. However, as these studies focus only on certain disciplines, there remains a gap in the literature concerning a clear framework of use across academic programmes. As a result, the issue of efficiently integrating games and simulations in the educational process is often up to the instructor’s discretion. Accordingly, the aim of this paper is to develop a framework to allow educators across disciplines to better understand the advantages and draw backs of games and simulations specific to their pedagogical goals.

Rationale of the study

The researchers set out to examine the effectiveness of games and simulations in the learning experience, and immediately encounter the first challenge, which relates to a lack of clear empirical evidence on the issue (Farrington, 2011 ). The scientific field is too extensive and requires further investigation. Furthermore, there is currently no formal policy framework or guidelines recommended by governments or educational institutions on the adoption of games and simulations in education. This is the case for many countries in Europe, the US, and Australia, where it is the responsibility of the instructor or institution to incorporate games into the curriculum.

The main motivation for the current review lies in the fact that games are already, to a certain degree, integrated into educational systems to achieve a variety of learning outcomes (Connolly, 2012 ), yet a comprehensive policy is still lacking. In this paper, the first step was an attempt to conceptualize the terms “game” and “simulations”. Although the two terms are neither wholly synonymous, or completely differentiated, in the main body of this review, the focus will be on lumping them together and perceiving them as points across a multidimensional continuum (Aldrich, 2009 ; Renken, 2016 ), since these educational technologies are consolidated under the umbrella of an interactive virtual environment in digital education.

A primary aim is to identify studies concentrating on the use of games and simulations for learning purposes, and to analyse the results by comparing them to prior studies’ findings. Two research questions guide the review analysis: a) How can the best practices/methods for designing and incorporating games and simulations in student learning be identified? b) How can games/simulations enhance Higher Education?

The major difference between the current review and the previous reviews in the field is the conceptualization of the terms “games and simulations”, which acts as an umbrella for further typologies. In other words, the researchers include more genres of games and simulations in their systematic review, compared to the other literature reviews. In addition, the researchers’ intention is to focus on the impacts of games and simulations on learning outcomes. The researchers don’t focus only on the cognitive outcomes, which is the most obvious and common topic among other researchers but, simultaneously, they analyze behavioural and affective effects as well. Furthermore, most of the previous reviews focus on the impacts of games and simulations on the learning process of certain subjects (e.g. Science, Business, Nursing, etc.), whereas this study expands research in a wide spectrum of academic disciplines and subjects. Overall, the current study offers a systematic review that opens new areas for further discussion, highlighting that collaborative learning, teamwork and students’ engagement also play a significant role for a successful learning process.

Conceptualising games and simulations

In recent years, the interest in examining game use in higher education has increased. This includes educational games (Çankaya & Karamete, 2009 ), digital game-based learning (DGBL) (Yang, 2012 ), and applied games (van Roessel & van Mastrigt-Ide, 2011 ). In addition, scholars, sometimes, include interactive exercises (Mueller, 2003 ), video games (Biddiss & Irwin, 2010 ), or even expand to next generation video games (Bausch, 2008 ), in the category of games. With respect to web-based games, the technological platforms that implement digital game code include computers and consoles (Salen & Zimmerman, 2004 ). They can run on a web browser on mobile phones and other mobile gaming devices (Willoughby, 2008 ) (e.g., tablets).

Despite the abundance of game types, there is a lack of clear, shared definitions and terminology among scholars and educators, which has led to “terminological ambiguity” (Klabbers, 2009 ). Nevertheless, the need for shared terminology remains when discussing the different forms of games and simulations in higher education. Although academics and game developers may use varying taxonomy to categorise games, the majority broadly agree on the following seven genres (Gros, 2007 ):

Action games: response-based video games.

Adventure games: the player solves problems to progress through levels within a virtual world.

Fighting games: these involve fighting with computer-controlled characters or those controlled by other players.

Role-playing games: players assume the roles of fictional characters.

Simulations: games modelled after natural or man-made systems or phenomena, in which players have to achieve pre-specified goals.

Sports games: these are based on different kinds of sports.

Strategy games: these recreate historical scenes or fictional scenarios, in which players must devise an appropriate strategy to achieve the goal.

In recent years, several well-designed empirical studies investigating the effects of serious games on learning outcomes have been published. Sawyer refers to serious games as those games produced by the video game industry that have a substantial connection to the acquisition of knowledge (Sawyer, 2002 ). Zyda ( 2005 ) expands Sawyer’s definition, adding that serious games are games whose primary purpose is not entertainment, enjoyment or fun. Serious games, educational gaming, as well as virtual worlds developed for educational purposes reveal the potential of these technologies to engage and motivate beyond leisure activities (Anderson et al., 2009 ). At the same time, there is extensive literature exploring the potential learning benefits offered by game-based learning (GBL), which can be defined as the use of game-based technology to deliver, support, and enhance teaching, learning, assessment, and evaluation (Connolly, 2007 ).

• Simulations

Simulations create a scenario-based environment, where students interact to apply previous knowledge and practical skills to real-world problems, also allowing teachers to reach their own goals, as well (Andreu-Andrés & García-Casas, 2011 ; García-Carbonell & Watts, 2012 ; Angelini, 2015 ). During scenario-based training, the player acquires important skills, such as interpersonal communication, teamwork, leadership, decision-making, task prioritising and stress management (Flanagan, 2004 ). The practical scenario may be carried out individually or within a team (Robertson et al., 2009 ), leading to collaboration and knowledge sharing.

With the explosion of Web 2.0 technology, increased opportunities to engage with technological applications in a collaborative and participatory way have emerged, promoting information access, shared ideas, knowledge exchange, and content production (McLoughlin & Lee, 2008 ). Digital simulations, which engage students in the interactive, authentic, and self-driven acquisition of knowledge, are being adopted in higher education. Connolly and Stansfield ( 2006 ) define game-based e-learning as a digital approach which delivers, supports, and enhances teaching, learning, assessment, and evaluation. Game-based e-learning is differentiated from GBL, which tends to cover both computer and non-computer games.

Delivery platforms are an essential aspect for game designers when creating and distributing games and simulations (e.g. computer, video, online, mobile, 3D, etc.). Designers must pay attention to characteristics such as the technical challenges, modules and techniques associated with the game design, the players involved in gaming, and the teaching modes (e.g. single, multi-player, collaborative, synchronous, etc.). This study examines the diverse curricular areas and learning objectives each game intends to access. The above-mentioned game classification is presented below (Fig. 1 ).

Classification of games and simulations

The main difference between games and simulations is the following: games are tools which are artificial and pedagogical; they include conflict, rules, and predetermined goals, whereas simulations are dynamic tools, representing reality, claiming fidelity, accuracy, and validity (Sauve, 2007 ).

Previously conducted reviews/meta-analyses on games and simulations in higher education

To establish a context , the researchers, initially, examined the relevant literature on the effectiveness of all types of games and simulations in learning outcomes. Many papers are analysed and summarised as follows, providing useful guidance for this study.

Through their systematic review, Tsekleves et al. ( 2014 ) provide insight into the barriers and benefits of using serious games in education. (Regarding benefits, the authors catalogue: achievement and rewards, interactivity and feedback, motivation and competition, playfulness and problem-based learning, collaborative learning, progression and repetition, as well as realism and immersion. Finally, they propose some guidelines to help stakeholders better implement serious games in education. Similarly, Bellotti, ( 2013 ) suggest useful guidelines for the performance assessment of serious games. Following user performance assessments, they offer an overview on the effectiveness of serious games in relation to learning outcomes. Results reveal the effectiveness of serious games in motivating and achieving learning goals, the importance of providing appropriate user feedback, while emphasizing that new types of games are best deployed through proper instructor guidance. Moreover, they stress aspects they consider important, such as performance assessment with a view to fostering adaptivity, as well as personalisation, and meeting needs on an individual basis (e.g. learning styles, information provision rates, feedback, etc.).

The instructor’s role is also outlined by Lameras et al. ( 2016 ) who provide conceptual and empirical evidence on the manner in which learning attributes and game mechanics should be designed and incorporated by faculty, specifically with a view to fully integrate these into lesson plans and the learning process as a whole. Games allow practitioners to quickly come to grips with the way in which learning activities, outcomes, feedback and roles may vary, as well as to enhance the in-game learning experience. Similarly, the systematic review of 64 articles by de Smale, ( 2015 ) concludes that there is a positive or neutral relationship between the use of simulations and games and learning achievement. The researchers arrive at three recurring conditions for the successful use of simulations and games: the specificity of the game, its integration in the course, and the role of a guiding instructor, which are all conditions in line with Bellotti et al. ( 2013 )‘s results.

Young et al. ( 2012 ) choose 39 articles that meet the inclusion criteria related to video games and academic achievement, concentrating on the use of traditional games versus video games for educational purposes. The studies are categorised by subject, namely History, Mathematics, Physical Education, Science, and Languages. Results indicate that there exists limited evidence of the benefits of including education games in the traditional classroom environments, a finding which is contrary to the aforementioned studies. Smetana and Bell ( 2012 ) examine computer simulations to support instruction and learning in Science. In their comparative study between computer games and traditional games, they conclude that computer games can be as effective, if not more so, than traditional games in promoting knowledge, developing procedural skills and facilitating conceptual change. To integrate them properly as supplementary elements (Rajan, 2013 ), games require the adoption of high-quality support structures, student participation, as well the promotion of cognitive and metacognitive skills. This finding contradicts the study carried out by Girard, ( 2013 ). This study treats video games as serious games but considers their effectiveness as a controversial issue, finding that only few games result in improved learning, while others have no positive effect on knowledge and skills acquisition, when compared to more traditional methods of teaching.

In contrast, in their meta-analysis, Clark et al. ( 2015 ) systematically review articles to study the detailed effects of digital games on learning outcomes, concluding that games are important in supporting productive learning and highlighting the significant role of gaming design beyond its medium. Prior to this review, but running along the same lines, Backlund and Hendrix ( 2013 ), in their meta-analysis reported positive outcomes in learning when using serious games in the educational process. Wouters, ( 2013 ) performing meta-analytic techniques, used comparisons as well, to investigate whether serious games are more effective and more motivating than conventional instructional methods. They found higher effectiveness in terms of learning and retention, but less motivation compared to traditional instructional methods. Indeed, serious games tend to be more effective if regarded as a supplement to other instructional methods, and involve students in groups and multiple training sessions.

These findings are compatible with those in the survey conducted by Rutten, ( 2012 ), which focuses on implementing games as laboratory activities, concluding that simulations have gained a prominent position in classrooms by enhancing the teacher’s repertoire, either as a supplement to traditional teaching methods or as a partial replacement of the curriculum. Nevertheless, they stress that the acquisition of laboratory skills cannot be wholly conducted via simulations. However, in areas where simulations have been widely accepted as a training tool, simulations can play a significant role in making lab activities more effective when offered as pre-lab training. Fu, ( 2016 ), through a systematic literature review, identify the multi-dimensional positive impact of serious games in business education, with the most frequent outcomes being knowledge acquisition and content understanding. The study also confirms that GBL and serious games can influence player engagement, perpetual and cognitive skills and social or soft skills. The affective and motivational outcomes are examined in entertainment games, games for learning and serious games, which reflects the trend of using gaming elements as both a medium of entertainment as well as a mode of learning. Ritzhaupt, ( 2014 ) produce meta-analysis based on 73 articles, demonstrating that achievement measures (e.g., standardised test scores) are the most commonly investigated, while the second most frequent is affective measures (e.g., usability or attitudes towards technology) followed by behavioural measures (e.g., task behaviour).

Merchant, ( 2014 ), via a meta-analysis, compare the effectiveness of games, simulations and virtual worlds in improving learning outcomes. Findings indicate that playing games individually enhance student performance more than playing collaboratively. Nonetheless, the researchers claim that there is no statistically significant difference between the effects of individual and cooperative instructional modules regarding simulations. Student learning outcomes deteriorate after repeated measures, since after spending a certain amount of time playing games, the learning outcome gains start to diminish. On the contrary, Shin, ( 2015 ), through meta-analysis, aim to identify the effects of patient simulation in nursing education. They find significant post-intervention improvements in various domains for participants who receive simulation education compared to the control groups, thus leading to the conclusion that simulations are more effective than traditional learning methods, enhancing the player’s psychomotor, affective, and cognitive skills. In their work, simulations provide students with authentic clinical situations, allowing them to practice and experience in realistic and safe environments.

Connolly et al. ( 2012 ) develop a multi-dimensional approach to categorising games and offer a review of 129 papers on computer games and serious games, explicitly targeting cognitive, behavioural, affective and motivational impacts, as well as engagement. The most frequent outcomes are knowledge acquisition and content understanding, as well as affective and motivational outcomes. Gegenfurtner, ( 2014 ) in their meta-analysis of the cognitive domain, examine how design elements in simulation-based settings affect self-efficacy and transfer of learning. They conclude that gathering feedback post-training, as opposed to during the process, results in higher estimates of self-efficacy and transfer of learning.

Researchers also look at games and simulations from a theoretical perspective. Li and Tsai ( 2013 ), examine the theoretical background and models employed in the study of games and simulations. They focus principally on the theories of cognitivism, constructivism, enactivism, and the socio-cultural perspective. Results indicate that although cognitivism and constructivism are the major theoretical foundations employed by game-based science learning researchers, enactivism and the socio-cultural perspective are the emerging theoretical paradigms drawing increasing attention in this field. This literature review indicates an increasing recognition of the effectiveness of digital games in promoting scientific knowledge and concept learning, while giving lesser importance to facilitation of problem-solving skills, exploring outcomes from the viewpoint of scientific processes, affect, engagement and socio-contextual learning. This view is echoed by other researchers, such as Warren, ( 2016 ), who systematically review and demonstrate the effectiveness of simulation games on satisfaction, knowledge, attitudes, skills and learning outcomes within nurse practitioner programmes. After comparing online simulation-based learning with traditional lectures, they find an increase in student knowledge and confidence when using simulation games. Peterson ( 2010 ) also performs a meta-analysis, examining the use of computerised games and simulations in language education from a psycholinguistic and socio-cultural viewpoint. Results show valuable opportunities for effective language learning, confirming that games are beneficial in helping students learn another language.

Sitzmann ( 2011 ), using interactive cognitive complexity theory, offers a comparative review on the instructional effectiveness of computer simulations. To perform the review, she examines three affective outcomes (motivation, effort, and self-efficacy), one behavioural (effort), two cognitive (declarative knowledge and retention), and two skill-based learning outcomes (procedural knowledge and transfer). She concludes that, post-training, simulation-trained learners demonstrate higher self-efficacy and procedural knowledge. Furthermore, she highlights the significance of using specific methods to improve simulation learning, namely, integration of game use within an instructional programme, high level of learner activity, no gaming time limit, and adopting the simulation game as a supplement to other methods, which is inconsistent with Wouters et al.’s survey ( 2013 ). Hsu et al. ( 2012 ) provide a cross-analysed content analysis agreeing with the previous researchers that topics such as “Motivation, Perceptions and Attitudes” are of utmost importance.

In a recent review of business literature, Carenys and Moya ( 2016 ) discuss the impact of digital game-based learning (DGBL) on students. They examine DGBL both from a theoretical point of view and on a practical basis through three stages: a) the evaluation of digital games in the preparatory stage, b) specifying which research has been deemed appropriate for DGBL deployment, and c) the learning outcomes (cognitive, behavioural, affective, and multi-dimensional) that can be attained through digital games. This study moved current research forward in understanding the effectiveness of digital games and advanced the use of digital games in the classroom.

A variety of meta-analyses and systematic reviews have examined the implementation of games and simulations in the learning process, either as a main course element or as a supplement to conventional lectures, illustrating the ever increasing interest of researchers in this promising field.

Synthesis of previous reviews/meta-analyses

After studying the previous reviews, it is evident that the most commonly referred games in past reviews are digital and computerized games (Sitzmann, 2011 ; Young et al., 2012 ; Smetana & Bell, 2012 ; Girard et al., 2013 ; Merchant et al., 2014 ; Clark et al., 2015 ; Carenys & Moya, 2016 ; Warren et al., 2016 ). The technological revolution and the invasion of Internet in Higher Education urge students to build digital and collaborative skills for the twenty-first century through gaming. Also, the emergence of a participatory culture in education spurs researchers to get involved with digital games and simulations. Other games mentioned are serious games and their impact on the learning process (Connolly et al., 2012 ; Bellotti et al., 2013 ; Backlund & Hendrix, 2013 ; Wouters et al., 2013 ; Tsekleves et al., 2014 ; Fu et al., 2016 ). The researchers refer to serious games since they are basically considered as games with educational orientation and not with just entertaining ones.

Another important element we have identified is whether games should be fully or partially integrated into the learning process. Most of the researchers agree that games should be treated mainly as supplementary elements (Sitzmann, 2011 ) since full integration requires high-quality mechanisms, students’ engagement, and instructors’ support. In other cases, the integration of games in the curriculum could either function as a supplement to existing teaching techniques or as a partial substitute for traditional teaching methods (Rutten et al., 2012 ). Moreover, games could even be fully integrated for achieving better learning outcomes (Lameras et al., 2016 ) because games add diversity in educational teaching modules. Nevertheless, the integration of games depends on instructors’ contribution and the way they design and incorporate games in their teaching. This means that instructors should be equipped with knowledge and experience, and be aware of providing guidance to students as regards the proper way of playing games.

The beneficial contribution of game-based learning is broadly identified by the majority of previous reviewers, especially regarding cognitive outcomes. Results indicate that games can be as effective as traditional learning modes, revealing their effectiveness in promoting knowledge acquisition (Smetana & Bell, 2012 ; Backlund & Hendrix, 2013 ; Clark et al., 2015 ; Warren et al., 2016 ), as well as content understanding and concept learning (Connoly et al., 2012 ; Li & Tsai, 2013 ; Fu et al., 2016 ). Additionally, students achieve their learning goals through playfulness and problem-based learning (Tsekleves et al., 2014 ), thus leading to self-efficacy and transfer of learning (Gegenfurtner et al., 2014 ).

Another substantial impact emerged is the effectiveness of games not only in the cognitive domain but also in the affective and behavioural domains (Ritzhaupt et al., 2014 ; Shin et al., 2015 ; Tsekleves et al., 2014 ; Fu et al., 2016 ; Carenys & Moya, 2016 ). The affective domain is thoroughly discussed by the reviewers. In particular, games influence students’ motivation, engagement, and satisfaction of the game-based learning. Regarding behavioural outcomes, few reviews have been conducted, showing that games offer a plethora of opportunities for collaborative learning, enhance interactivity and feedback among players, and develop social and soft skills as well. Some other studies contradict these findings, in a way that they do not reveal positive effects of games (Young et al., 2012 ; Girard et al., 2013 ; Merchant et al., 2014 ), or reveal a rather neutral effect (de Smale et al., 2015 ). In these reviews, games and simulations appear to have some or no positive effects on knowledge and skills acquisition when comparing with traditional instructional methods.

Research method

Research selection.

The authors developed a pre-defined review protocol to answer the research questions, specifically aimed at minimising researcher bias. The literature review was carried out between July and October 2016 and followed the design stages described below.

The reviewed papers are identified through keywords in referenced electronic databases, such as Google Scholar, Web of Science, ERIC, PsycInfo, PsycArticles Fulltext Search, InterDok, ProQuest, Scopus, BEI, and SearchPlus. The keywords for learning outcomes are a combination of the term games or simulations paired with the term higher education , employing the Boolean operator “AND”. Additional keywords for learning outcomes are learning objectives, learning goals, learning objectives and effects . Keywords for platform and delivery methods include computer-based, web-based, digital, virtual, online, and technology. Keywords for games and simulations are educational games, business simulations, role-playing simulations, game-based learning, video games, and serious games . Moreover, the Boolean operator “OR” is employed to combine all these keywords. The study sets the broadest range of keywords, so as not to limit the scope of related articles.

Furthermore, the researchers conducted a comprehensive database search in bibliographic indices for the data selection. The search is related to a variety of scientific fields of study, including Education, Psychology, Information Technology, Management, and other scientific areas (e.g., Engineering, STEM, Health, etc).

Assessment and extraction

The dataset consists of journal articles referring to games, simulations or learning in their title and/or abstract. The researchers piloted and evaluated their selection criteria based on prior studies. The study selection process was conducted in two separate phases: a) the researchers, working independently, initially, and, subsequently, together, screened the titles and abstracts for inclusion criteria, and b) in the event of disagreement or insufficient information, they carried out a thorough consideration of the body of the articles (i.e. methodology and results), again independently, first, and, then, together, resulting in consensus. Then, whether to include the text or not was discussed, based on pre-determined criteria. The inclusion criteria used are as follows:

Only empirical articles across a variety of study designs may be included, so as to achieve rich data.

The participants should be over the age of 18 (e.g., students in higher education, college students, instructors, etc.)

Articles that provide an evaluation of student learning outcomes (via the use of games for pedagogical purposes) may also be included.

The resources should, mainly, consist of journal articles and conference papers, which, due to the peer review process, ensure a high quality of material to examine. Existing meta-analysis and systematic literature reviews should be included as well, in order to cross-validate the review findings.

The articles should be available in either English or French.

The articles should adhere to the objective of the study and the definition of the terms games and simulations as pedagogical applications.

Studies containing samples from higher education institutions should also be included. Conversely, research on the effects of games in primary or secondary education should be discarded.

The review should include games and simulations used in traditional, as well as in online environments.

Only peer-reviewed journal articles published between 2010 and 2016 should be included, as the intention is to include the most current research.

Several exclusion criteria, listed below, were also applied in this study:

Non-empirical studies or studies which solely describe the design of a learning environment.

Participants who are younger than 18 years old.

Non-GBL tools and entertainment games.

Book chapters -not only are books difficult to search for on databases, they are, also, hard to access as full texts. Additionally, books are not always subject to the same peer review process as scientific articles. Dissertations, theses, editorials, book reviews and reports are also excluded for similar reasons.

Articles that cannot be accessed as full texts are excluded.

Articles that do not match the research objectives.

Research focused on types of education other than higher education.

As mentioned above, articles published before 2010.

The following figure illustrates the inclusion and exclusion criteria (Fig. 2 ):

The inclusion and exclusion criteria

Application of these criteria resulted in an initial dataset, yielding 8859 studies, published between 2010 and 2016. The selected papers are derived from 67 academic journals representing a variety of disciplines. Most papers come from the scientific journal “Computers & Education”, while “British Journal of Educational Technology”, and “Simulation & Gaming” were the next two journals appearing with the most frequency. In the final stage, several meetings were organized between researchers to discuss the findings, and to decide on presentation.

The results show a steady increase in published papers discussing games from 2012 onwards. After systematically reviewing their abstracts, a final shortlist of 628 potential full text articles emerged. Two hundred and seventeen out of the 628 were excluded, primarily, due to undesirable focus (e.g. theoretical scenarios for using simulations in education). For each of the remaining 411 studies, the researchers identified and recorded some basic themes, for example, types of learning outcomes, effect or impact of game and simulation methods on learning goals, participants and settings, research questions, research methodology and results. Of these, 123 papers, which are found to contribute data, are selected for the review, whereas the remaining 288 articles are excluded, due to the fact that they are written in a language that the researchers do not understand, or because they are focused on a field other than higher education. The majority of these articles are published in scientific journals or conference proceedings, whereas 25 studies are either meta-analyses and/or systematic reviews. An outline of the entire review is depicted in the following figure (Fig. 3 ):

Research review methodological scheme

Data analysis and synthesis

The identified articles are analysed using a qualitative content analysis technique, which leads to a coding scheme, including a main category, three sub-categories and several associated topics related to the main categories. The researchers unanimously agree upon the coding that emerged from the analysis of the reviewed papers. To ensure inter-rater reliability (p) with respect to the quality of article coding procedures, a small random sample ( n  = 20) of the selected articles is coded in duplicate. The calculated reliability exceeds 93%, which is a high quality of agreement across coding categories. Furthermore, a review of mixed-methodology studies provides high-quality evidence, due to a combination of quantitative and qualitative elements in terms of methodological triangulation.

The researchers examined the studies from varying viewpoints. Firstly, they analysed the data set characteristics, such as the continent on which the studies are conducted, the subject discipline, the methodological research design, the types of games and simulations identified, and the time period in which the majority of the studies took place. The emphasis is on the analysis, measures, and design of the quantitative methodology (experimental, quasi-experimental, pre-test, post-test surveys, etc.), as well as the qualitative methods used in the reviewed surveys.

To sum up, the review studies are selected through a systematic process with pre-arranged criteria. There is no intended bias applied to the selected studies, and although the majority of studies come from Europe, this is simply the result of the systematic selection process.

Data set characteristics

When analysing the data, the researchers came across some interesting characteristics. Other than the meta-analytic studies and review research, the locations of the remaining surveys are as follows: 33% conducted in Europe, 22% in Asia, and 18% in the USA, whereas 24% of the articles do not directly mention a location (Fig. 4 ). Most of the articles come from the USA, the UK, and the Netherlands.

Continets where studies are conducted

With respect to genre, there is a diverse representation of games and simulations. The most prominent game genre identified in the relevant literature seems to be simulation games in general, that is to say, virtual/online games or simulations, computer-based learning, role-playing games, serious games, and business simulation games. This representation is illustrated below (Fig. 5 ):

Representation of the game genres

With respect to the busiest publication period, the majority of studies that meet the inclusion criteria were published between 2013 and 2016, as shown in the following bar chart (Fig. 6 ). This finding demonstrates a notable trend amongst researchers discussing the topic of games and simulations in recent years, due to increased awareness of the use of technological games in higher education.

Years of published articles

The data also represents a wide range of subject areas. Some cover multiple areas, for example Engineering, Management, Science, Law, Social Sciences and Humanities (Tao et al., 2015 ), or even just two areas, such as Biology and Computer Sciences (Yang & Chang, 2013 ), while others refer to only one academic discipline. The subject areas are sorted into larger categories, with the most common area being Business Management and Marketing. The results are shown in the figure below (Fig. 7 ):

Subject disciplene

The reviewed articles include data from 99 samples and 20,406 participants, which is a considerably large grouping. The population tested in the literature review ranges from 5 participants in small qualitative studies (Ke et al., 2015 ) to 5071 participants in extensive quantitative quasi-experimental research (Lu et al., 2014 ). Most of the participants are young undergraduate, graduate or post-graduate students, and faculty members. The studies consistently indicate a good gender balance in participants. In some studies, there is both student and faculty participation (Kapralos et al., 2011 ; Felicia, 2011 ; Hess & Gunter, 2013 ; Hämäläinen & Oksanen, 2014 ; Beuk, 2015 ; Crocco, 2016 ), whereas in others, only instructors are chosen as participants (Tanner, 2012 ; Badea, 2015 ; Franciosi, 2016 ). On the whole, most studies use students as participants.

Procedures and research methodologies

Most studies use either an experimental or a quasi-experimental design employing a pre-test and/or a post-test evaluation, with four using only a pre-test questionnaire, and six using only post-test evaluations. The effects of games and simulations on learning outcomes are measured through calculating the difference between pre-test and post-test scores of the experimental or quasi-experimental design. More specifically, the researchers compare the increases in scores between control and experimental groups to evaluate the effectiveness of using the tested games and simulations. The studies include longitudinal surveys (e.g. Hainey, 2011 ) conducted for a specified number of years, whereas others are comparative studies (e.g., Boeker, 2013 ; Poikela, 2015 ).

Researchers use quantitative methods in the majority of studies (68.6%), while13.1% use qualitative methodology. Some studies follow a mixed research methodology (nearly 18.2%), providing pragmatic perceptions and methodological triangulation of the results. The measures utilized in quantitative studies include knowledge questionnaires, as well as academic, evaluation, and cognitive tests, while in qualitative studies the methods used include interviews, case studies, observations and focus groups.

The studies portray a variety of time periods spent playing games and simulations: some of the participants interact with games over a single session, while others are involved in the gaming process for several weeks or even months (e.g., Yang & Chang, 2013 ; Woo, 2014 ). The studies include multi-player games (e.g., Silvia, 2012 ; Yin, 2013 ), as well as single-player games.

Learning outcomes of games and simulations

In the present review, keeping in mind the aforementioned research questions (p.3), the researchers break down their findings in relation to the learning outcomes of games and simulations into three categories, namely cognitive, behavioural, and affective outcomes. A map of the emerging concepts, which will be further discussed, is illustrated below (Fig. 8 ):

Learning outcomes of Games/Simulations

Cognitive outcomes

Many reviewed studies discuss the impact of GBL activities in learner knowledge acquisition and conceptual understanding (Hainey et al., 2011 ; Connolly et al., 2012 ; Fu et al., 2016 ; Geithner & Menzel, 2016 ). There has been an impact evaluation across subject disciplines, such as Computer Science (Strycker, 2016 ), Engineering (Chaves et al., 2015 ), Physics (Adams, 2016 ), Medicine (Dankbaar, 2016 ), Nursing (Sarabia-Cobo, 2016 ), Management (Geithner & Menzel, 2016 ), Political Sciences (Jones & Bursens, 2015 ), Education (Ke, 2015 ), Languages (Franciosi, 2016 ), and Social Sciences (Cózar-Gutiérrez & Sáez-López, 2016 ).

Knowledge acquisition

Cognitive outcomes refer “to the knowledge structures relevant to perceiving games as artefacts for linking knowledge-oriented activities with cognitive outcomes” (Lameras et al., 2016 , p. 10). Tasks framed as games and simulations are deployed to develop a diverse range of cognitive skills, such as deep learning (Vos & Brennan, 2010 ; Young et al., 2012 ; Erhel & Jamet, 2013 ; Crocco et al., 2016 ), critical thinking and scientific reasoning (Beckem & Watkins, 2012 ; Halpern et al., 2012 ; Ahmad, 2013 ), action-directed learning (Lu et al., 2014 ), transformative learning (Kleinheskel, 2014 ), decision-making (Tiwari, 2014 ), knowledge acquisition and content understanding (Terzidou, 2012 ; Elias, 2014 ; Fu et al., 2016 ), spatial abilities (Adams et al., 2016 ), and problem solving (Liu, 2011 ; Lancaster, 2014 ).

The effect of games and simulations on learning remains a controversial issue amongst researchers in the field, as it will be further confirmed in this article. Some reviewed studies indicate improved learning, while others show no positive effect on knowledge and skill acquisition compared to traditional learning methods. The value of simulations can be examined from the perspective of content change as discussed in Kovalic and Kuo’s study ( 2012 ). Simulations are directly linked to the course content and students are given the opportunity to apply and better understand theoretical concepts. Additionally, simulations provide an environment in which students can experiment with different strategies, adopt different roles, and take charge of their own decisions by assuming responsibility. The latter issue is discussed at length by Liu et al. ( 2011 ), who find that, when solving problems, students are more likely to learn via playing a game than via a traditional learning experience.

Serious gaming, especially given the context of enthusiastic students, has proved to be an effective training method in domains such as medical education, for example, in clinical decision-making and patient interaction (de Wit-Zuurendonk & Oei, 2011 ). Similarly, Kleinheskel ( 2014 ) illustrates the importance of designing self-reflective simulating activities for nursing students, and aligning such design with cognitive outcomes. When students self-reflect on simulated clinical experiences, they add to their existing knowledge, and apply new knowledge to transformative learning. Poikela et al. ( 2015 ), in a simulated nursing procedure, compare a computer-based simulation with a lecture to examine the meaningful learning students may achieve via the two teaching methods. They conclude that students who participate in the computer simulation are more likely to report meaningful learning outcomes than those taking the lecture, due to the strong presence of reflection-based activities and metacognitive themes. Similar results are present in Chen, ( 2015 ), survey in which both solitary players and collaborative groups achieve equally positive learning outcomes in a game. Students significantly improve judging by their pre- and post-test assessments, which indicates that the gaming experience affects their overall performance, and, most likely, promotes conceptual understanding. Moreover, collaborative GBL allows students to re-construct and co-construct knowledge, thus encouraging problem-solving through peer discussion.

Challenging games enhance participant performance (Wang & Chen, 2010 ; Gold, 2016 ). This finding is supported by von Wangenheim, ( 2012 ), who analyse the cognitive dimension of an educational game focusing on memory, understanding and conceptual application. The validity of micro-simulation games is identified by participants in Lukosch, ( 2016 ), research who evaluate a specific microgame as an excellent instrument for enhancing situated and experiential learning by transferring knowledge to an actual situation at the workplace. The results comply with those of Riemer and Schrader ( 2015 ), where the application of comprehension and transfer of knowledge are best achieved using simulations.

Furthermore, the impact of game-based learning on learning performance has been observed by numerous researchers across diverse subjects, as reported above (Zacharia & Olympiou, 2011 ; Rutten et al., 2012 ; Beckem & Watkins, 2012 ; Boeker et al., 2013 ; Shin et al., 2015 ; Hou, 2015 ; Chen et al., 2015 ; Tao et al., 2015 ). For instance, Divjak and Tomić ( 2011 ) provide evidence that computer games impact mathematical learning, revealing the positive effect of games on student learning outcomes. Reviews by Young et al. ( 2012 ) confirm the effectiveness of using videogames on History, Languages, and Physical Education. The analysis of four experimental virtual conditions in pre- and post-test assessments reveal that virtual experimentation promotes conceptual understanding in Physics students (Zacharia & Olympiou, 2011 ). A 3D visualisation and simulation laboratory activity on protein structure is more effective than traditional instruction modules, as described in White, ( 2010 ), research resulting in students preferring to work with visualized simulations.

Simulation games also positively affect clinical practice situations. “The Ward”, a simulation game in Stanley and Latimer’s ( 2011 ) research proves to be an enjoyable and valuable learning tool in addressing clinical skill practice, nursing practice knowledge, critical thinking and decision-making. Vos and Brennan ( 2010 ) highlight the effectiveness of marketing simulation games, where students perceive simulations as an enjoyable learning approach, contributing to decision-making, as well as other valuable knowledge and skills, a finding consistent with Tiwari et al. ( 2014 ) survey. Swanson et al. ( 2011 ) created a rubric to measure the effectiveness of teaching strategies in nursing education. The experimental post-test assessment survey aims to evaluate the effects of three teaching strategies on the outcome of performance and retention of intervention activities, student satisfaction, self-confidence and practical educational preferences. Results reveal significantly higher retention scores compared to the first assessment, indicating that high scores in the improved rubric are related to the interactivity of the simulation scenario.

Nevertheless, it should not be taken for granted that students consistently prefer virtual learning settings to more traditional face-to-face environments (Hummel et al., 2011 ). Serious games concerning cognitive perceptions show varying results. For example, simulations are shown to support the comprehension and application of knowledge, albeit less effectively than quizzes and adventures (Riemer & Schrader, 2015 ). In Fu et al. ( 2016 ) review, despite GBL providing a motivating and enjoyable experience, there is a lack of strong evidence to show that games lead to effective learning outcomes. In some cases, there is inconsistency in student views regarding the integration of online games as a positive learning method (Bolliger, 2015 ). Similar views are supported by some researchers, who acknowledge students’ and educators’ hesitation towards virtual simulations and serious games, but they insist on the inclusion of games into course material, and on instructors’ familiarization with their use (Kapralos et al., 2011 ).

Perceptual skills

Other studies confirm the power of games and simulations in developing cognition abilities, especially in the instances of virtual simulations enhancing complex cognitive skills (Helle et al., 2011 ; Siewiorek, 2013 ), such as self-assessment (Arias Aranda, 2010 ), or higher-order thinking (Crocco et al., 2016 ). These are meta-cognitive skills, regarded as essential elements of in-depth learning. The incorporation of game mechanisms into simulations is widely recognised by researchers as beneficial, especially regarding laboratory tasks, where simulation scenarios urge students towards problem-solving and, reflection, thus achieving metacognitive outcomes (Hou & Li, 2014 ; Hou, 2015 ). Kikot, ( 2013 ) concur with the above researchers, stating that students perceive simulation-based learning (SBL) environments positively when asked to achieve dynamic learning outcomes, including thinking, interpreting, and associative skills.

Silvia ( 2012 ) also references cognitive and metacognitive outcomes derived from a multi-role simulation. The simulation helps students apply the concepts they learn in class by connecting the theoretical issues with real-world situations, thus developing their analytical skills, and through comparing different viewpoints, which leads to enhanced critical thinking. Students use the interactive nature of simulations to develop arguments, make judgements and evaluate situations. More importantly, simulations encourage students to develop self-awareness. Similarly, Cela-Ranilla, ( 2014 ) conducted a study in which students display a tendency to perform better in analytical work, such as monitoring, planning and assessment rather than in action-based work. Wouters et al. ( 2013 ), on the other hand, find serious games to be more effective in terms of learning and retention.

Learners can also actively participate in a web-based simulation to facilitate immersion and reflection, leading to deeper understanding of the content (Helle et al., 2011 ). A simulation framework can facilitate learning in terms of flow experience and learning strategies. Indeed, in a study conducted by Li, Cheng, and Liu ( 2013 ), the framework helps students lacking background knowledge to balance challenge and skill perceptions, while for students with average to advanced levels of knowledge, it facilitates the learning experience by either reducing the challenge perception or promoting the skill perception. Along the same lines, Pasin and Giroux ( 2011 ), analyse the mistakes students make in simulations using an empirical prototype. Results show that, although simple decision-making skills are easily acquired through conventional teaching methods, simulation games are useful tools for mastering managerial skills, such as complex and dynamic decision-making. Lin and Tu ( 2012 ) also confirm that simulations enable students to train themselves in decision-making.

Instructors’ engagement

Students are challenged to develop interpersonal, analytical and creative skills, discouraging absenteeism, feelings of boredom and reluctance, leading to academic achievement. However, simulations not only exhibit positive effects in the learning experience of the student, but, also, do so for instructors, as well, in the context of teaching experience. For academics, simulations raise the level of performance, encouraging students to be more alert and attentive during class activities (Navidad, 2013 ), and thus to achieve better learning outcomes. In this vein, instructors are urged to design simulations to be as challenging as possible to stimulate student interest in interacting with the simulation as well as with their peers. Felicia ( 2011 ) denotes that instructors agree with students in acknowledging the educational benefits of video games, such as an understanding of difficult concepts, improvement of spatial awareness and analytical skills, critical thinking, and problem-solving strategies. To enable them to do so, instructors emphasize the importance of clearly expressed learning goals to guide students when using simulations in an online instructional technology course (Kovalik & Kuo, 2012 ).

Even setting aside the potential learning benefits derived from participation in GBL, a stronger connection between games and curricula remains to be forged, as well as the application of more dynamic academic challenges, so as to better adapt to the knowledge of diverse learners (Pløhn, 2013 ). Following such reasoning, as indicated in the literature, faculty plays a key role in achieving learning goals via the use of games and simulations. The instructor role correlates with the demand for abstract learning concepts. In their meta-analysis, Wouters and Van Oostendorp ( 2013 ) show how instructors, acting in a facilitating and supporting role, can foster learning, particularly in selecting and discussing new information and where higher order skills are involved in the learning outcomes. Similarly, instructors can monitor student behaviour and evaluate not only the capabilities, but also the attitudes of tomorrow’s higher education managers during the decision-making process. Rutten et al. ( 2012 ) focus in their literature review on the level of instructional support in GBL, and suggest that a pedagogical framework for the application of computer simulations in education requires a corresponding integration of the educator’s role.

Behavioural outcomes

Behavioural objectives for higher education students refer to the enhancement of teamwork and improvement in relational abilities (Ranchhod, 2014 ), as well as stronger organisational skills, adaptability and the ability to resolve conflicts (Vos & Brennan, 2010 ).

Social skills/teamwork

Simulation games are often seen as powerful tools in promoting teamwork and team dynamics (Stanley & Latimer, 2011 ; Tiwari et al., 2014 ; Lin, 2016 ; Wang, 2016 ), collaboration (Hanning, 2012 ), social and emotional skills (Ahmad et al., 2013 ), and other soft skills, including project management, self-reflection, and leadership skills (Siewiorek, 2012 ; Wang et al., 2016 ), which are acquired through a reality-based scenarios with action-oriented activities (Geithner & Menzel, 2016 ).

In a Spanish management course, simulations enabled students to build pivotal capacities, such as management abilities and team working to enable the success of future managers (Arias Aranda et al., 2010 ). A computer simulation at a university in Taiwan led to comparatively higher learning gains against traditional teaching through collaborative laboratory activities (Shieh, 2010 ), by facilitating students to carry out more active learning and improving their conceptual understanding. Simulation scenarios provide improved social and communication skills, which lead to the enhancement of student knowledge (Sarabia-Cobo et al., 2016 ).

Additionally, collaboration is considered an essential element in the learning process (Elias, 2014 ). The findings of Hummel et al. ( 2011 ) reveal that serious online games improve the quality of learning when it comes to problem-based situations in the workplace by using active collaboration. For this reason, faculty members are urged to create learning environments to support active participation by students in the educational process. Moreover, according to the constructivist approach, the instructor’s role is a significant factor in empowering groups to construct knowledge in a collaborative manner (Hämäläinen & Oksanen, 2014 ). The instructors engage higher education students in the process of formulating hypotheses, interpreting context, providing explanations, and describing observations, by designing and implementing a collaborative and interactive GBL environment. In Yin et al.’s study ( 2013 ), students react positively to participatory simulations, due to the belief that the system helps them advance their conceptual understanding effectively through scaffolding, discussion, and reflection. Participants in Cózar-Gutiérrez and Sáez-López’s study ( 2016 ), while stating that video games are non-essential tools in an educational context, nevertheless, value GBL as an immersive environment that facilitates increased activity and student engagement.

Teamwork, however, seems to be a controversial issue in Costa, ( 2014 ) which evaluates improvement of knowledge sharing. Some learners consider teamwork as a means to facilitate decision making in a game, while others express dissatisfaction due to their peers, be it the latter’s reluctance to take on responsibility or poor negotiation capabilities. Research by Bolliger et al. ( 2015 ) similarly indicates that some learners remain hesitant, as they feel the use of games may actually decrease opportunities for communication with peers and instructors. Merchant et al. ( 2014 ) conclude that student performance is enhanced when playing individually rather than in a group.

Interaction and feedback

In GBL methods, meaningful feedback is a key factor in students achieving the objectives, as well as in being encouraged to reflect on misunderstandings and to transfer learning to new educational contexts (Swanson et al., 2011 ). In the current study, the scope is to investigate learner-learner interaction and social feedback through game mechanics. Higher education students evaluate games and simulations focusing on behavioural change and improvement of interactive abilities. The computer game DELIVER! for example, is evaluated very positively by students due to its focus on active student participation and overall positive impact on social interaction (von Wangenheim et al., 2012 ). Simulations provide visual feedback, encouraging active exploration of the student’s own understanding, enabling a move beyond knowing-in action and beginning to reflect-on and in-action during training, resulting in the contextual application of prior knowledge (Söderström, 2014 ). Real-time feedback in simulation games enables students to clearly define the objectives and expectations in the interactive environment, leading to a reduction in anxiety and uncertainty, thus encouraging better performance (Nkhoma et al., 2014 ).

The literature extensively documents the interaction between behavioural outcomes, learning performance and communication especially in Online Distance Learning (ODL). Indeed, regular feedback on student performance during DGBL facilitates deep learning (Erhel & Jamet, 2013 ). A survey conducted by Chen, ( 2010 ) shows that online games can be social and interactive technologies, helping students form friendships with their peers and providing multiple types of interaction.

Ke et al. ( 2015 ) stress the importance of player interaction, indicating that the inherent interaction between players and their gaming-situated learning environment supplies structured challenges and feedback. Huang, ( 2010 ) share the same view, confirming that, due to the necessity of receiving feedback from peers and the game itself, increased interaction opportunities arise in game-play, adding that interaction is a decisive factor in the construction of knowledge (Seng & Yatim, 2014 ). In a survey conducted by Denholm et al. ( 2012 ), students report improved team working through the use of serious games. They attribute this to receiving feedback, and stressing that even conflict is often considered valuable as it brings diverse views to the fore.

To conclude, the main body of literature explores the impact of games and simulations on learning outcomes on the behavioural level, especially when students are involved in interactive and participatory simulation tasks. The majority of studies reveal a positive effect on behavioural outcomes, concluding that students benefit from appropriate feedback, and reflection through game-based communication activities.

Affective outcomes

Many studies highlight the affective outcomes of using games and simulations in the learning process. The majority of them includes student engagement (Auman, 2011 ; Hainey et al., 2011 ; Lin & Tu, 2012 ; Kikot et al., 2013 ; Lu et al., 2014 ; Ke et al., 2015 ), motivation (Liu et al., 2011 ; Liao & Wang, 2011 ; Costa et al., 2014 ; Lukosch et al., 2016 ), and satisfaction (Cvetić et al., 2013 ; Dzeng, 2014 ; Lancaster, 2014 ; Sarabia-Cobo et al., 2016 ).

Motivation and engagement

Engagement and motivation are major factors in enhancing higher education learning objectives (Connolly et al., 2012 ; Erhel & Jamet, 2013 ; Ke et al., 2015 ; Nadolny & Halabi, 2015 ). Motivation is considered a central factor in the majority of reviewed studies (Felicia, 2011 ; Ljungkvist & Mozelius, 2012 ; von Wangenheim et al., 2012 ; Bellotti et al., 2013 ; Hannig et al., 2013 ; Ahmad et al., 2013 ; Pløhn, 2013 ; Li et al., 2013 ; Denholm et al., 2012 ; Dzeng et al., 2014 ; Lancaster, 2014 ; Ariffin et al., 2014 ; Bolliger et al., 2015 ; Cózar-Gutiérrez, & Sáez-López, 2016 ; Dankbaar et al., 2016 ; Fu et al., 2016 ). Some results suggest the effectiveness of GBL in motivating and achieving learning goals can be found at the lower levels of Bloom’s taxonomy (e.g. Connolly et al., 2012 ). In the context of digital SBL environments, other motivational dimensions are highlighted, such as self-efficacy (Sitzmann, 2011 ), in conjunction with the transfer of learning (Gegenfurtner et al., 2014 ).

Motivation is a combination of elements such as attention, relevance, confidence, and satisfaction, which can increase germane cognitive loads. Chang, ( 2010 ) examine the effects of motivation in an instructional simulation game, called SIMPLE. According to the post-game evaluation, student motivation comes from peer learning and user cooperation. Moreover, when instructors teach strategy, this enhances student motivation and engagement, encouraging acceptance of the game, and leading to stronger interest in course-directed learning. Thus, teachers should create a flexible learning environment, giving due consideration to peer interaction, learning motivation, pedagogical support and encouragement to help students develop their autonomy and retain an interest in learning.

Another important element contributing to affective outcomes is challenge. Hainey et al. ( 2011 ) find the presence of a challenge to be the top ranked motivation for online game players, while recognition is the lowest ranked motivation regardless of gender or amount of players in the game. Gamers in a multiplayer environment tend to report competition, cooperation, recognition, fantasy and curiosity when playing games, while online players experience challenge, cooperation, recognition and control. By contrast, fanatical computer game players experience disappointment and a lack of challenge, as they tend to value the technical aspect over the challenges presented by game play. In Hess and Gunter’s survey ( 2013 ), students in a game-based course are motivated, because of the positive social interaction they experience while playing the game; this intrinsic motivation is positively correlated to student performance. Computer games can thus be seen as a learning tool motivating players to acquire many competences. Connolly et al. ( 2012 ) share the same view, seeing the role of challenge as a predictive factor with respect to game engagement and achievement. Similarly, in Ke et al.’s study ( 2015 ), the game-play actions include optimal challenge expectation for the user. These results can also be seen in Badea ( 2015 ), who concludes that the majority of participants in her study acknowledge the highly motivating quality of games, which are complemented by the relaxed class atmosphere when games are used.

However, despite the benefits reaped from the implementation of games and simulations concerning affective outcomes, some researchers underline that motivation is not always related to GBL, emphasizing cases where students who use games in solitary or collaborative environments experience no significant difference in terms of learning motivation (Chen et al., 2015 ). There are indeed cases where serious games are no more motivating than conventional instructional methods (Wouters et al., 2013 ). In Cela-Ranilla et al.’s survey ( 2014 ), despite the suitability of the 3D simulation environment, students do not feel highly motivated or particularly engaged, mostly because they prefer analysis to actions in the particular learning process.

Faculty role

The benefits of a pedagogical shift from a teacher-focused and lecture-based classroom to a student-centred, active-learning environment through the adoption of simulation-based strategies to achieve engagement are relevant to both students and instructors (Auman, 2011 ). There is a progression in student emotion from uncertainty and nervousness to satisfaction and excitement within the gaming experience. Auman ( 2011 ), as an instructor, provides a positive description: she is drawn in by student enthusiasm, her interest in the material is reinvigorated, she feels empowered in her teaching, and ready to guide her class. In this context, it’s easy to see how instructors ought to play a significant role in motivating and engaging students to achieve learning goals. De Porres and Livingston ( 2016 ) concur with Auman ( 2011 ), as their study also indicates increased levels of excitement in doctoral students studying Computer Science, when evaluated in a post-test intervention.

Faculty acting as motivators are key in engaging students in the learning process, working to ensure focus on pre-existing knowledge, as well as to transfer knowledge to game settings (Lameras et al., 2016 ), to reward students for their effort, and support them by providing continuous guidance and pathways for further consideration. The quality of the teacher/facilitator has a strong influence on the learning satisfaction of the students. Also, instructors should facilitate and engage students via in-game discussion forums to help overcome misconceptions, and to lead the game-based learning. The way instructors interact, facilitate and motivate students to construct GBL experiences depends on the design stage, particularly on the way games are incorporated into the curriculum in a traditional course (Wouters et al., 2013 ). This is because motivation exhibits a significant correlation with cognitive and skill performance (Woo, 2014 ). In research conducted by Franciosi ( 2016 ), despite faculty acknowledging the beneficial impact of games on student motivation, they nevertheless, remain doubtful about the effectiveness of games in learning outcomes, thus resulting in neutral attitudes. Interestingly, although instructors perceive simulations as engaging learning technologies, they do not however consider them superior to traditional teaching methods (Tanner et al., 2012 ).

Another aspect, less frequently discussed in the relevant literature, is students’ performing self-assessments with regard to effective learning, as seen in Jones and Bursens study ( 2015 ). This ability is supported by constructivism, since simulations are developed in an active learning environment, where faculty act more as facilitators rather than as instructors and students are provided with feedback to carry out their self-assessments.

Attitudes and satisfaction

A vital element in achieving learning goals is the relationship between motivational processing and the outcome processing (satisfaction), especially in an online instructional game, as seen in the experiment carried out by Huang et al. ( 2010 ). There seems to be a significant relation between these two variables, which suggests that designers of DGBL need to consider extrinsic rewards to achieve motivational development and satisfaction. Learning satisfaction is strongly correlated with student motivation and attitude towards GBL before the game, with actual enjoyment and effort during the game, as well as with the quality of the teacher/facilitator (Mayer, 2013 ). Specifically, students with a higher level of inner motivation and positive attitude towards GBL are more likely to have higher learning expectations, and to experience more satisfaction in their GBL participation.

In general, most studies report that students develop a positive attitude toward the pedagogical adoption of games and simulations in education (Divjak & Tomić, 2011 ; Bekebrede, 2011 ; Ibrahim et al., 2011 ; Beckem & Watkins, 2012 ; Tanner et al., 2012 ; von Wangenheim et al., 2012 ; Halpern et al., 2012 ; Terzidou et al., 2012 ; Hanning et al., 2013 ; Giovanello, 2013 ; Cvetić et al., 2013 ; Kovalik & Kuo, 2012 ; Li & Tsai, 2013 ; Hainey et al., 2011 ; Boeker et al., 2013 ; Nkhoma et al., 2014 ; Costa et al., 2014 ; Chaves et al., 2015 ; Riemer & Schrader, 2015 ; Angelini, 2016 ; Geithner & Menzel, 2016 ). The participants in Dudzinski et al. ( 2013 ) respond positively towards a serious web-based game, describing the experience as interesting, stimulating and helpful, as well as a valuable addition to their pharmacy curriculum. Other students perceive simulation games as fun, but not particularly useful as an instructional method compared to lectures, and about equally useful as case discussions (Beuk, 2015 ). In another study, the majority of students show a positive attitude towards games, positing that they make subjects more fun and provide more opportunities for learning (Ibrahim et al., 2011 ). This finding is consistent with Bekebrede et al. ( 2011 ) on the perceptions of Dutch students belonging to the “net generation”, who have been raised with technology-based games. Data reveals student preference towards active, collaborative and technology-rich learning via digital games that bring added value to the educational process.

For students, satisfaction is a deciding factor in their decision to continue using such learning methods (Liao & Wang, 2011 ; Liao, 2015 ). Terzidou et al. ( 2012 ) discuss affective outcomes, especially the way interviewees feel before and after their participation in the game. Prior to participating, the interviewees report feelings of entertainment, fascination, and satisfaction before their participation in the game, which increase after use, indicating that participants find the use of 3D virtual game appealing.

Chen et al. ( 2010 ) reveal that the majority of students show negative feelings about online gaming. Shieh et al.’s ( 2010 ) mixed methodology research reveals that experimental groups show positive attitudes toward an innovative learning environment and outperform the control groups (in conventional classes). Some studies depict either neutral effects (Rajan et al., 2013 ; Beuk, 2015 ; Bolliger et al., 2015 ; Dankbaar et al., 2016 ; Strycker, 2016 ) or negative attitudes towards game use in the learning experience (Jiménez-Munguía & Luna-Reyes, 2012 ). Students experience more anxiety and boredom during conventional courses, which acts as an impediment to acquiring substantial problem-solving skills. The educational benefits of GBL are particularly apparent in subjects over which students report greater anxiety, where it can be proven that increased enjoyment levels correlate positively with improvements in deep learning and higher-order thinking (Crocco et al., 2016 ). Liarokapis, ( 2010 ) show Computer Science students evaluating a serious online game, and finding it a valuable pedagogical tool, which is both useful and entertaining.

Genre/familiarity issues

Students achieving high scores respond more positively to online games compared to low achieving students. Regarding genre perceptions, male students express more enthusiasm towards digital gaming than female students, or at least spend more time playing computer games compared to girls (Hainey et al., 2011 ). This may be due to the fact that boys tend to be more familiar with computers and web-based technologies. Girls may choose to avoid digital game-based learning methods, due to their negative preconceptions about gaming, preventing them from harnessing the positive aspects of online gaming (Chen et al., 2010 ). These studies indicate a difference in perception based on gender when engaging in DGBL environments. However, research by Riemer and Schrader ( 2015 ) concluded that female students reported a more positive attitude and perception of affective quality compared to the male students. Also, high assessment scores in web-based games depend on the professional experience of the players. Unexpectedly, in Dzeng et al.’s experimental survey ( 2014 ), despite the high test scores achieved in both web-based and paper-based games, students without work experience achieve the highest post-test scores, probably because they are more familiar with using technological tools. The experiments in Erhel and Jamet’s study ( 2013 ) indicate that serious games promote learning and motivation, provided they include features that prompt learners to actively process the educational content.

To sum up, games and simulations lead to improved affective outcomes for university students such as attitudes, motivation, emotional involvement, self-efficacy and satisfaction. A growing body of literature supports the positive attitude shown by students towards games and simulations, as they consider them essential instructional tools that provide motivation and engagement in an active learning environment.

Research interest in the incorporation of games and simulations in higher education is constantly developing (Girard et al., 2013 ). The pedagogical shift, from lecture-centred to student-centred environments and the increasing use of games as innovative learning technologies, calls for a transformation in higher education. In this respect, games and simulations are expected to play a significant role in the learning process. In the present study, the focus is on the positive effects of games and simulations on university students’ learning outcomes. The reviewed papers are diverse in terms of research objectives, theoretical background, methodological avenues adopted, game genres, scientific domain or delivery platform, and various perspectives concerning cognitive, behavioural and affective outcomes employed. Many articles ( n  = 123) are identified, providing either empirical results or offering meta-analytic evidence.

There seems to be a lack of shared definitions or taxonomy necessary for a common classification, which, therefore, results in terminological ambiguity (Klabbers, 2009 ). The majority of GBL researchers compare the effectiveness of implementing web-based learning games to conventional instructional options (Shin et al., 2015 ).

Mapping the results, empirical evidence is identified with respect to cognitive learning outcomes including knowledge acquisition, conceptual application, content understanding and action-directed learning. Games and simulations are educational interventions, which create a supportive environment in which students may acquire knowledge across subjects and disciplines. Students have the opportunity to better understand theoretical concepts, provided that games are used as a supplement in traditional lecture-based courses. Additionally, simulations are often perceived as enjoyable learning tools, which require active and collaborative participation and contribute to the improvement of critical thinking and reasoning, higher-order- and metacognitive thinking. Simulations provide students the opportunity to observe the outcomes of their actions, and take responsibility for decision-making via problem-solving competencies, thus leading to a more active, transformative and experiential reception of knowledge.

Another important finding is that simulations have positive effects on both students and instructors. Positive outcomes exist when instructors set achievable learning goals, interact with students promoting knowledge, support, facilitate, and motivate them to construct new game-based knowledge (Kovalik & Kuo, 2012 ; Lameras et al., 2016 ). Instructors are encouraged to design games and simulations in order to make students fully aware of game activities, providing all the while continuous instructional guidance. These results generally confirm the findings from prior systematic reviews and meta-analyses. However, findings diverge slightly in Young et al.’s survey ( 2012 ), who claim that there is limited or no evidence about the effective implementation of games in the lecture-based curriculum.

This review also covers behavioural outcomes, mainly the development of social, emotional, and collaborative skills, helping students to foster strong relationships with peers, empowering them to collaborate and work in groups more efficiently, become organised, adapt to new tasks, and resolve emerging conflicts. Furthermore, reality-based scenarios and action-oriented game activities promote fruitful interactions and meaningful feedback, which leads to collaborative construction of knowledge. Overall, digital games and simulations urge students to interact not only with the game, but with their instructors and co-players as well. These results have been extensively covered in the literature review, with the majority of researchers agreeing with the current study’s results, confirming the positive effects of games and simulations on the behavioural level of learning outcomes (Bellotti et al., 2013 ; Tsekleves et al., 2014 ; Fu et al., 2016 ; Carenys & Moya, 2016 ).

However, although most reviews acknowledge the positive effects of games in behavioural outcomes, some reviewed studies contradict these positive findings, claiming that teamwork is a controversial issue when it comes to the improvement of knowledge sharing. The use of games seems to decrease opportunities for peer interaction and communication with instructors (Bolliger et al., 2015 ), whereas playing individually is sometimes considered better than working in a team (Merchant et al., 2014 ). Also, in some cases, games and simulations through collaborative activities distract students and hinder learning (Dankbaar et al., 2016 ).

The current review makes a significant contribution by investigating the affective outcomes when incorporating games and simulations in the curriculum, especially motivational and engagement outcomes, emotional development, satisfaction, attitude, emotion, self-assessment, and self-efficacy. Results show that games and simulations motivate, engage and promote effective learning goals by providing opportunities for learners to actively experience, practice, interact, and reflect in a collaborative, game-based, and learner-centred setting. The measures evaluating student attitudes reveal an increasingly positive trend towards games and simulations, especially in post-interventions (Bekebrede et al., 2011 ; Giovanello et al., 2013 ; Costa et al., 2014 ; Angelini, 2016 ; Geithner & Menzel, 2016 ).

To this end, there has been a purposeful highlighting of the instructor’s role as facilitator and motivator in this literature review. Through in-game activities and extended discussion, instructors promote student interaction and help them overcome the lack of understanding of content curriculum and achieve better learning outcomes. The literature also stresses the role of emotional development, which facilitates improvement of learning outcomes. Specifically, there seems to be a progression in student emotion, from negative feelings including uncertainty, anxiety, nervousness, and disappointment during pre-intervention, to positive feelings of satisfaction, confidence, excitement, enjoyment, effort, fascination, and enthusiasm during in-game and post-game interventions (Huang et al., 2010 ; Hummel et al., 2011 ; Liao & Wang, 2011 ; Terzidou et al., 2012 ; Woo, 2014 ; Liao et al., 2015 ).

Most of the pre-existing evidence is compatible with the findings of this systematic review (Sitzmann, 2011 ; Connolly et al., 2012 ; Wouters et al., 2013 ; Ritzhaupt et al., 2014 ; Gegenfurtner et al., 2014 ; Shin et al., 2015 ; Lameras et al., 2016 ; Carenys & Moya, 2016 ; Fu et al., 2016 ; Warren et al., 2016 ). Nevertheless, one study indicates that the overall positive perception of students depends on the different forms of games (Riemer & Schader, 2015 ), namely, simulations promote a less positive effect compared to quizzes and adventures. Some other studies diverge further in their findings, indicating either neutral (Rajan et al., 2013 ; Strycker, 2016 ; Franciosi, 2016 ) or negative student attitudes towards the use of games (Chen et al., 2010 ; Jiménez-Munguía & Luna-Reyes, 2012 ). Also, there are limited results on the effect of games on student self-efficacy, with one study demonstrating moderate post-training self-efficacy (Sitzmann, 2011 ).

Comparing the findings of the current study with the findings of previous systematic reviews and meta-analyses leads to an interesting discussion. The results of the present review illustrate that the majority of the revised articles focus on different genres of games and simulations. The mostly represented genres are virtual/online games and simulations since they can enhance learning in certain disciplines, such as Computer Studies. This finding is in agreement with most of the previous reviews (e.g. Clark et al., 2015 ; Carenys & Moya, 2016 ; Warren et al., 2016 ). Also, simulation games are found to be popular in this review, due to the fact that they are implemented in authentic learning environments, namely in Health Sciences and Biology. Also, in this study, a great representation of role - playing games and business simulation games are obviously resulted from the previous articles, due to the fact that they are implemented in specific academic disciplines, such as Business Management and Marketing. Nevertheless, in this review, serious games are not represented as much as in other reviews (e.g.Tsekleves et al., 2014 ; Fu et al., 2016 ).

Additionally, this study concentrates on the positive effects of games and simulations on learning outcomes, a finding that is compatible with previous reviews (e.g. Bellotti et al., 2013 ; Lameras et al., 2016 ; Clark et al., 2015 . This review confirms that games and simulations contribute to cognitive learning outcomes, including knowledge acquisition, conceptual application, content understanding, and action-directed learning. Other previous reviewers echoed these findings (Smetana & Bell, 2012 ; Shin et al., 2015 ; Wouters et al., 2013 ; Fu et al., 2016 ) emphasizing the important role of games in knowledge acquisition and content understanding. It has been illustrated that university students benefit from the incorporation of games into the learning process, if used as a supplement in traditional lectures, a finding that complies with other reviews (Sitzmann, 2011 ; Wouters et al., 2013 ). However, simulations’ implementation is influenced by instructors’ guidance and motivation, as these factors encourage faculty to design simulations to achieve learning outcomes.

This review also sheds light on behavioural outcomes of using games in instructional design. The emphasis is on the positive effects, namely the development of social and soft skills, emotional skills, the empowerment of collaboration with peers, and the promotion of interaction and feedback, findings that are in line with past reviews (Shin et al., 2015 ; Carenys & Moya, 2016 ). Despite the positive behavioural effects of utilizing games, some reviews find collaboration and teamwork as a hindrance for learning. The application of games seems to decrease peer interaction and communication with faculty, whereas in Merchant et al.’s review ( 2014 ), playing individually is more preferable than playing collaboratively. The current review concludes by highlighting the affective outcomes, and the emphasis is given on motivational and engaging factors that lead to emotional development, satisfaction, self-efficacy and self-assessment, findings that are also documented in other reviews (Sitzmann, 2011 ; Hsu et al., 2012 ; Tsekleves et al., 2014 ).

To conclude, this review discusses the multitude of surveys on the cognitive, behavioural, and affective outcomes related to the use of playing games and simulations in higher education. The multi-dimensional analysis of the empirical data provides a framework for understanding the major outcomes of GBL. Despite the significant benefits in learning outcomes highlighted in this paper, the high cost of designing games and simulations is still a significant challenge. To overcome this cost barrier, governments, researchers, instructors, and game designers should collaborate to find affordable solutions, for enabling the development of games and simulations. Since this review does not concern itself with advanced aspects of learning, the focus should next turn to a metacognitive-oriented survey, which will study the promotion of metacognitive skills in students, such as self-regulation, self-reflection, self-awareness, evaluation, planning, building on the ideas of others, debating, and so forth.

Future research

Considering the above discussion points, and the importance of games and simulations as derived from the relevant literature, some suggested avenues for future research are as follows:

Researchers should focus on applying the relevant theoretical frameworks, such as cognitivism, constructivism, and socio-cultural perspectives to cognitive, behavioural and affective outcomes, respectively.

More research should be conducted investigating gender issues with respect to the effectiveness of games on developmental aspects of behaviour, such as scaffolding and immersion, to counteract the present gap in the existing literature.

Comparative surveys should be included with a design focused on different target groups (adult students, or K-12 students in laboratory conditions).

Evaluation models via a mixed-method design are encouraged, especially to investigate how game designers could tailor game designs to applying different learning preferences and styles.

University instructors should take a more active role in the alignment of games with the curriculum ensuring that games and simulations are implemented in a blended learning module (face-to-face, online material, etc.), or even acting as games masters, scaffolding virtual experiences to university learners.

Faculty should design games with a view to multiplayer cooperation to achieve effectiveness in learning outcomes. Students should also be involved as co-designers, recommending innovative ideas and radical approaches in an effort to meet their own needs. An innovative approach is the adoption of metagames (Young et al., 2012 ), which consist of additional learning resources (blogs, wikis, etc.) encouraging collaboration between players.

This study makes a significant contribution to research, since no other literature review or meta-analysis has been conducted so far investigating educational and web-based games and simulations with such an extensive subject and discipline coverage in higher education. Today’s demand for student-centred teaching methods to develop highly qualified learners, capable of learning in an active and collaborative environment, calls for the deployment of game-based activities and simulations that will enable them to face the challenges of the dawning era.

Adams, D. M., Pilegard, C., & Mayer, R. E. (2016). Evaluating the cognitive consequences of playing portal for a short duration. Journal of Educational Computing Research, 54 (2), 173–195.

Article   Google Scholar  

Ahmad, S. M. S., Fauzi, N. F. M., Hashim, A. A., & Zainon, W. M. N. W. (2013). A study on the effectiveness of computer games in teaching and learning. International Journal of Advanced Studies in Computers, Science and Engineering, 2 (1), 1.

Google Scholar  

Aldrich, C. (2009). Virtual worlds, simulations, and games for education: A unifying view. Innovate: Journal of Online Education, 5 (5), 1.

MathSciNet   Google Scholar  

Anderson, E. F., McLoughlin, L., Liarokapis, F., Peters, C., Petridis, P., & Freitas, S. D. (2009). Serious games in cultural heritage. In M. Ashley & F. Liarokapis (Eds.), VAST 2009: 10th International Symposium on Virtual Reality, Archaeology and Cultural Heritage 22-25 September 2009 (pp. 29–48). St. Julians, Malta: Eurographics Association.

Andreu Andrés, M. A., & García Casas, M. (2011). Perceptions of gaming as experiential learning by engineering students. International Journal of Engineering Education, 27 (4), 795–804.

Angelini, M. L. (2016). Integration of the pedagogical models “simulation” and “flipped classroom” in teacher instruction. SAGE Open, 6 (1). doi: 10.1177/2158244016636430 .

Angelini, M. L., García-Carbonell, A., & Martínez-Alzamora, N. (2015). Estudio cuantitativo discreto sobre la simulación telemática en el aprendizaje del ingles. [Quantitative study about telematic simulation in learning]. RIE Revista Iberoamericana de Educación, 69 (2), 51–68.

Arias Aranda, D., Haro Domiguez, C., & Romerosa Martinez, M. M. (2010). An innovative approach to the learning process in management: The use of simulators in higher education. REVISTA DE EDUCACION, 353 , 333–334.

Ariffin, M., Oxley, A., & Sulaiman, S. (2014). Evaluating game-based learning effectiveness in higher education. Procedia - Social and Behavioral Sciences, 123 , 20–27 http://dx.doi.org/10.1016/j.sbspro.2014.01.1393 . Accessed 11 Sept 2016.

Auman, C. (2011). Using simulation games to increase student and instructor engagement. College Teaching, 59 (4), 154–161.

Backlund, P., & Hendrix, M. (2013). Educational games-are they worth the effort? A literature survey of the effectiveness of serious games. In Games and virtual worlds for serious applications (VS-GAMES), 2013 5th international conference on (pp. 1-8). IEEE.

Badea, M. (2015). English classes and effectiveness of games in higher education. Journal Plus Education, 13 (2), 79–85.

Bausch, L., Beran, J., Cahanes, S., & Krug, L. (2008). Physiological responses while playing Nintendo Wii sports. Journal of Undergraduate Kinesiology Research, 3 , 19–25.

Beckem, J. M., & Watkins, M. (2012). Bringing life to learning: Immersive experiential learning simulations for online and blended courses. Journal of Asynchronous Learning Networks, 16 (5), 61–70.

Bekebrede, G., Warmelink, H. J. G., & Mayer, I. S. (2011). Reviewing the need for gaming in education to accommodate the net generation. Computers & Education, 57 (2), 1521–1529.

Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Interaction, 2013 , 1.

Beuk, F. (2015). Sales simulation games student and instructor perceptions. Journal of Marketing Education , 1-13 (2015). Doi: 0273475315604686.

Biddiss, E., & Irwin, J. (2010). Active video games to promote physical activity in children and youth. Archives of Pediatrics and Adolescent Medicine, 164 , 664–672.

Boeker M, Andel., P, Vach, W., & Frankenschmidt, A. (2013). Game-based E-learning is more effective than a conventional instructional method: A randomized controlled trial with third-year medical students. PloS One 8(12): e82328. doi: 10.1371/journal.pone.0082328 .

Bolliger, D. U., Mills, D., White, J., & Kohyama, M. (2015). Japanese students’ perceptions of digital game use for English-language learning in higher education. Journal of Educational Computing Research, 53 (3), 384–408 0735633115600806.

Buckless, F. A., Krawczyk, K., & Showalter, D. S. (2014). Using virtual worlds to simulate real-world audit procedures. Issues in Accounting Education, 29 (3), 389–417.

Çankaya, S., & Karamete, A. (2009). The effects of educational computer games on students’ attitudes towards mathematics course and educational computer games. Procedia-Social and Behavioral Sciences, 1 (1), 145–149.

Carenys, J., & Moya, S. (2016). Digital game-based learning in accounting and business education. Accounting Education , 25 (6), 598-651.

Cela-Ranilla, J. M., Esteve-Mon, F. M., Esteve-González, V., & Gisbert-Cervera, M. (2014). Developing self-management and teamwork using digital games in 3D simulations. Australasian Journal of Educational Technology, 30 (6), 634–651.

Chang, Y. C., Peng, H. Y., & Chao, H. C. (2010). Examining the effects of learning motivation and of course design in an instructional simulation game. Interactive Learning Environments, 18 (4), 319–339.

Chaves, R. O., von Wangenheim, C. G., Furtado, J. C. C., Oliveira, S. R. B., Santos, A., & Favero, E. L. (2015). Experimental evaluation of a serious game for teaching software process modeling. IEEE Transactions on Education, 58 (4), 289–296.

Chen, L., Chen, T.-L., & Liu, H.-K. J. (2010). Perception of young adults on online games: Implications for higher education. TOJET: The Turkish Online Journal of Educational Technology, 9 (3), 76–84.

Chen, C. H., Wang, K. C., & Lin, Y. H. (2015). The Comparison of solitary and collaborative modes of game-based learning on Students' science learning and motivation. Educational Technology & Society, 18 (2), 237–248.

Chiang, Y. T., Lin, S. S. J., Cheng, C. Y., & Liu, E. Z. F. (2011). Exploring online game players’ flow experiences and positive affect. The Turkish Online Journal of Educational Technology, 10 (1), 106–114.

Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2015). Digital games, design, and learning a systematic review and meta-analysis. Review of educational research, 86 (1), 79–122 0034654315582065.

Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of the empirical evidence on computer games and serious games. Computers & Education, 59 (2), 661–686.

Connolly, T., & Stansfield, M. (2006). Using games-based eLearning technologies in overcoming difficulties in teaching information systems. Journal of Information Technology Education, 5 (1), 459–476.

Connolly, T. M., Stansfield, M., & Hainey, T. (2007). An application of games-based learning within software engineering. British Journal of Educational Technology, 38 (3), 416–428.

Costa, G. J. M. D., Kikot, T., Fernandes, S., & Águas, P. (2014). Why use-centered game-based learning in higher education? The case of cesim simbrand. Journal of Spatial and Organizational Dynamics , 2 (3), 229-241.

Cózar-Gutiérrez, R., & Sáez-López, J.M. (2016). Game-based learning and gamification in initial teacher training in the social sciences: An experiment with MinecraftEdu. International Journal of Educational Technology in Higher Education , 13 (1), 1 (2016). doi: 10.1186/s41239-016-0003-4 .

Crocco, F., Offenholley, K., & Hernandez, C. (2016). A proof-of-concept study of game-based learning in higher education. Simulation & Gaming, 47 (4), 403-422 (2016). Doi: 1046878116632484.

Cvetić, B., Vasiljević, D., & Danilović, M. (2013). DRP game: New tool to enhance teaching and learning in logistics and supply chain management. In 1st Logistics International Conference (pp. 299-303).

Dankbaar, M. E. W., Alsma, J., Jansen, E. E. H., van Merrienboer, J. J. G., van Saase, J. L. C. M., & Schuit, S. C. E. (2016). An experimental study on the effects of a simulation game on students’ clinical cognitive skills and motivation. Advances in Health Sciences Education, 21 (3), 505–521.

DePorres, D., & Livingston, R. E. (2016). Launching new doctoral students: Embracing the Hero’s journey. Developments in Business Simulation and Experiential Learning, 43 (1), 121–128.

de Wit-Zuurendonk, L. D., & Oei, S. G. (2011). Serious gaming in women’s health care. BJOG: An International Journal of Obstetrics & Gynaecology, 118 (s3), 17–21.

Denholm, J., Protopsaltis, A., & de Freitas, S. (2012). The value of team-based mixed-reality (TBMR) games in higher education. In European Conference on Games Based Learning (p. 571). Academic Conferences international limited.

de Smale, S., Overmans, T., Jeuring, J., & van de Grint, L. (2015). The effect of simulations and games on learning objectives in tertiary education: A systematic review. In Games and Learning Alliance 4 th International Conference , GALA 2015 Rome.

Divjak, B., & Tomić, D. (2011). The impact of game-based learning on the achievement of learning goals and motivation for learning mathematics-literature review. Journal of Information and Organizational Sciences, 35 (1), 15–30.

Dudzinski, M., Greenhill, D., Kayyali, R., Nabhani, S., Philip, N., Caton, H., Ishtiaq, S., & Gatsinzi, F. (2013). The design and evaluation of a multiplayer serious game for pharmacy students. In European Conference on Games Based Learning (p. 140). Academic Conferences international limited.

Dzeng, R. J., Lin, K. Y., & Wang, P. R. (2014). Building a construction procurement negotiation training game model: Learning experiences and outcomes. British Journal of Educational Technology, 45 (6), 1115–1135.

Elias, A. (2014). Simulating the European Union: Reflections on module design. International Studies Perspectives, 15 (4), 407–422.

Erhel, S., & Jamet, E. (2013). Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness. Computers & Education, 67 , 156–167.

Farrington, J. (2011). From the research: Myths worth dispelling: Seriously, the game is up. Performance Improvement Quarterly, 24 , 105–110.

Felicia, P. (2011). Assessing how game-based learning is perceived in Irish education. In Proceedings of the 7th European Conference on Management Leadership and Governance: ECGBL 2011 (p. 168). Academic Conferences limited.

Flanagan, B., Nestel, D., & Joseph, M. (2004). Making patient safety the focus: Crisis resource management in the undergraduate curriculum. Medical Education, 38 (1), 56–66.

Franciosi, S. J. (2016). Acceptability of RPG simulators for foreign language training in Japanese higher education. Simulation & Gaming, 47 (1), 31–50 (2015). Doi: 1046878115608621.

Fu, K., Hainey, T., & Baxter, G. (2016). A systematic literature review to identify empirical evidence on the use of computer games in business education and training. In 10th European Conference on Games Based Learning: ECGBL 2016 (p. 232).

García Carbonell, A., & Watts Hooge, F. I. (2012). Investigación empírica del aprendizaje con simulación telemática. Revista Iberoamericana de Educación (versión digital)., 59 (3), 1–11.

Gegenfurtner, A., Quesada-Pallarès, C., & Knogler, M. (2014). Digital simulation-based training: A meta-analysis. British Journal of Educational Technology, 45 (6), 1097–1114.

Geithner, S., & Menzel, D. (2016). Effectiveness of learning through experience and reflection in a Project Management simulation. Simulation & Gaming, 47 (2), 228–256 (2016). Doi: 1046878115624312.

Giovanello, S. P., Kirk, J. A., & Kromer, M. K. (2013). Student perceptions of a role-playing simulation in an introductory international relations course. Journal of Political Science Education, 9 (2), 197–208.

Girard, C., Ecalle, J., & Magnan, A. (2013). Serious games as new educational tools: How effective are they? A meta-analysis of recent studies. Journal of Computer Assisted Learning, 29 (3), 207–219.

Gold, S. (2016). Design and effectiveness of a self-study pedagogical approach to using a simulation game in the classroom. Developments in Business Simulation and Experiential Learning, 43 (1).

Gómez, M. (2014). Ciencias Sociales y gamificación, ¿una pareja con futuro? In J. Pagès (Ed.), Una mirada al pasado y un proyecto de futuro: investigación e innovación en didáctica de las ciencias sociales (pp. 257–262). Barcelona: Universitat Autónoma de Barcelona; Asociación Universitaria de Profesores de Didáctica de las Ciencias Sociales.

Gros, B. (2007). Digital games in education: The Design of Games Based Learning Environments. Journal of Research on Technology in Education, 40 (1), 23–39.

Hainey, T., Connolly, T. M., Stansfield, M., & Boyle, E. A. (2011). Evaluation of a game to teach requirements collection and analysis in software engineering at tertiary education level. Computers & Education, 56 (1), 21–35.

Halpern, D. F., Millis, K., Graesser, A. C., Butler, H., Forsyth, C., & Cai, Z. (2012). Operation ARA: A computerized learning game that teaches critical thinking and scientific reasoning. Thinking Skills and Creativity, 7 (2), 93–100.

Hämäläinen, R., & Oksanen, K. (2014). Collaborative 3D learning games for future learning: Teachers’ instructional practices to enhance shared knowledge construction among students. Technology, Pedagogy and Education, 23 (1), 81–101.

Hannig, A., Kuth, N., Özman, M., Jonas, S., & Spreckelsen, C. (2012). eMedOffice: A web-based collaborative serious game for teaching optimal design of a medical practice. BMC Medical Education, 12 (1), 1.

Hannig, A., Lemos, M., Spreckelsen, C., Ohnesorge-Radtke, U., & Rafai, N. (2013). Skills-o-mat: Computer supported interactive motion-and game-based training in mixing alginate in dental education. Journal of Educational Computing Research, 48 (3), 315–343.

Helle, L., Nivala, M., Kronqvist, P., Gegenfurtner, A., Björk, P., & Säljö, R. (2011). Traditional microscopy instruction versus process-oriented virtual microscopy instruction: A naturalistic experiment with control group. Diagnostic Pathology, 6 (1), 1.

Hess, T., & Gunter, G. (2013). Serious game-based and nongame-based online courses: Learning experiences and outcomes. British Journal of Educational Technology, 44 (3), 372–385.

Hou, H. T. (2015). Integrating cluster and sequential analysis to explore learners’ flow and behavioral patterns in a simulation game with situated-learning context for science courses: A video-based process exploration. Computers in Human Behavior, 48 , 424–435.

Hou, H. T., & Li, M. C. (2014). Evaluating multiple aspects of a digital educational problem-solving-based adventure game. Computers in Human Behavior, 30 , 29–38.

Hsu, Y.-C., Ho, H. N. J., Tsai, C.-C., Hwang, G.-J., Chu, H.-C., Wang, C.-Y., & Chen, N.-S. (2012). Research trends in technology-based learning from 2000 to 2009: A content analysis of publications in selected journals. Educational Technology & Society, 15 (2), 354–370.

Huang, W. H., Huang, W. Y., & Tschopp, J. (2010). Sustaining iterative game playing processes in DGBL: The relationship between motivational processing and outcome processing. Computers & Education, 55 (2), 789–797.

Hummel, H. G., Van Houcke, J., Nadolski, R. J., Van der Hiele, T., Kurvers, H., & Löhr, A. (2011). Scripted collaboration in serious gaming for complex learning: Effects of multiple perspectives when acquiring water management skills. British Journal of Educational Technology. doi: 10.1111/j.1467-8535.2010.01122.x .

Ibrahim, R., Wahab, S., Yusoff, R. C. M., Khalil, K., Desaru, I. N. S. T. E. D. T., & Jaafar, A. (2011). Student perceptions of educational games in higher education: An empirical study. Issues in Information Systems, 12 (1), 120–133.

Jiménez-Munguía, M. C., & Luna-Reyes, L. F. (2012). Development of analytical abilities and collaborative learning assessment in undergraduate students through simulation games. In Electrical Communications and Computers (CONIELECOMP), 2012 22nd International Conference on (pp. 12–16). IEEE.

Jones, R., & Bursens, P. (2015). The effects of active learning environments: How simulations trigger affective learning. European Political Science, 14 (3), 254–265.

Kapralos, B., Hogan, M., Pribetic, A. I., & Dubrowski, A. (2011). Virtual simulations and serious games in a laptop-based university: Gauging faculty and student perceptions. Interactive Technology and Smart Education, 8 (2), 106–120.

Ke, F., Xie, K., & Xie, Y. (2015). Game-based learning engagement: A theory-and data-driven exploration. British Journal of Educational Technology. doi: 10.1111/bjet.12314 .

Kikot, T., Costa, G., Magalhães, R., & Fernandes, S. (2013). Simulation games as tools for integrative dynamic learning: The case of the management course at the University of Algarve. Procedia Technology, 9 , 11–21.

Klabbers, J. H. G. (2009). Terminological ambiguity game and simulation. Simulation & Gaming, 40 (4), 446–463.

Kleinheksel, A. J. (2014). Transformative learning through virtual patient simulations: Predicting critical student reflections. Clinical Simulation in Nursing, 10 (6), e301–e308.

Kovalik, C. L., & Kuo, C. L. (2012). Innovation Diffusion: Learner benefits and instructor insights with the Diffusion Simulation Game. Simulation & Gaming. 43 (6), 803-824.

Lameras, P., Arnab, S., Dunwell, I., Stewart, C., Clarke, S., & Petridis, P. (2016). Essential features of serious games design in higher education: Linking learning attributes to game mechanics. British Journal of Educational Technology. doi: 10.1111/bjet.12467 .

Lancaster, R. J. (2014). Serious game simulation as a teaching strategy in pharmacology. Clinical Simulation in Nursing, 10 (3), e129–e137.

Li, Z. Z., Cheng, Y. B., & Liu, C. C. (2013). A constructionism framework for designing game-like learning systems: Its effect on different learners. British Journal of Educational Technology, 44 (2), 208–224.

Li, M. C., & Tsai, C. C. (2013). Game-based learning in science education: A review of relevant research. Journal of Science Education and Technology, 22 (6), 877–898.

Liarokapis, F., Anderson, E. F., Oikonomou, A. (2010). Serious games for use in a higher education environment. In Proceedings of the Emerging Games Platforms, Technologies and Applications Workshop (EGPTA’10), 15th Int‟l Computer Games Conference: AI, Interactive Multimedia, Virtual Worlds and Serious Games Louisville, Kentucky, USA (pp. 28-31).

Liao, Y. W., Huang, Y. M., & Wang, Y. S. (2015). Factors affecting students’ continued usage intention toward business simulation games an empirical study. Journal of Educational Computing Research, 53 (2), 260–283.

Liao, Y.-W., & Wang, Y.-S. (2011). Investigating the factors affecting students’ continuance intention to use business simulation games in the context of digital learning. In International Conference on Innovation, Management and Service (pp. 119-124).

Lin, Y. L. (2016). Differences among different DGBLs learners. International Journal of Business and Management, 11 (1), 181–188.

Lin, Y. L., & Tu, Y. Z. (2012). The values of college students in business simulation game: A means-end chain approach. Computers & Education, 58 (4), 1160–1170.

Liu, C. C., Cheng, Y. B., & Huang, C. W. (2011). The effect of simulation games on the learning of computational problem solving. Computers & Education, 57 (3), 1907–1918.

Ljungkvist, P., & Mozelius, P. (2012). Educational games for self learning in introductory programming courses-a straightforward design approach with progression mechanisms. In Proceedings Of The 6th European Conference On Games Based Learning, ECGBL (pp. 285-293).

Lu, J., Hallinger, P., & Showanasai, P. (2014). Simulation-based learning in management education: A longitudinal quasi-experimental evaluation of instructional effectiveness. Journal of Management Development, 33 (3), 218–244.

Lukosch, H., Kurapati, S., Groen, D., & Verbraeck, A. (2016). Microgames for situated learning a case study in interdependent planning. Simulation & Gaming, 47 (3), 346–367 (2016). Doi: 1046878116635468.

Mayer, I., Warmelink, H., & Bekebrede, G. (2013). Learning in a game-based virtual environment: A comparative evaluation in higher education. European Journal of Engineering Education, 38 (1), 85–106.

McLoughlin, C., & Lee, M. J. W. (2008). The three P’s of pedagogy for the networked society: Personalization, participation, and productivity. International Journal of Teaching and Learning in Higher Education, 20 (1), 10–27.

Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis. Computers & Education, 70 , 29–40.

Mueller, F., Agamanolis, S., & Picard, R. (2003). Exertion interfaces: Sports over a distance for social bonding and fun. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 561–568). ACM.

Nadolny, L., & Halabi, A. (2015). Student participation and achievement in a large lecture course with game-based learning. Simulation & Gaming, 47 (1), 51–72 (2015). Doi: 1046878115620388.

Navidad, F. C. (2013). Students' devised classroom games-simulations: An innovative tool on mathematics achievement and motivation in nursing students. International Proceedings of Economics Development and Research, 60 , 14–18. doi: 10.7763/IPEDR .

Nkhoma, M., Calbeto, J., Sriratanaviriyakul, N., Muang, T., Ha Tran, Q., & Kim Cao, T. (2014). Towards an understanding of real-time continuous feedback from simulation games. Interactive Technology and Smart Education, 11 (1), 45–62.

Pasin, F., & Giroux, H. (2011). The impact of a simulation game on operations management education. Computers & Education, 57 (1), 1240–1254.

Peterson, M. (2010). Computerized games and simulations in computer-assisted language learning: A meta-anlysis of research. Simulation & Gaming, 41 (1), 72–93.

Pløhn, T. (2013). Nuclear mayhem-a pervasive game designed to support learning. In European Conference on Games Based Learning (p. 475). Academic Conferences international limited.

Poikela, P., Ruokamo, H., & Teräs, M. (2015). Comparison of meaningful learning characteristics in simulated nursing practice after traditional versus computer-based simulation method: A qualitative videography study. Nurse Education Today, 35 (2), 373–382.

Rajan, P., Raju, P., & Sankar, C.S. (2013). Serious Games to Improve Student Learning in Engineering Classes . Paper presented at 120 th ASEE Annual Conference & Exposition , 23-26 June, 2013, Atlanta, Georgia. https://peer.asee.org/22448 Accessed 2 Oct 2016.

Ranchhod, A., Gurău, C., Loukis, E., & Trivedi, R. (2014). Evaluating the educational effectiveness of simulation games: A value generation model. Information Sciences, 264 , 75–90.

Renken, M., Peffer, M., Otrel-Cass, K., Girault, I., & Chiocarriello, A. (2016). Computer simulations on a multidimensional continuum: A definition and examples. In Simulations as Scaffolds in Science Education (pp. 5-14). London: Springer. 

Riemer, V., & Schrader, C. (2015). Learning with quizzes, simulations, and adventures: Students' attitudes, perceptions and intentions to learn with different types of serious games. Computers & Education, 88 , 160–168.

Ritzhaupt, A., Poling, N., Frey, C., & Johnson, M. (2014). A synthesis on digital games in education: What the research literature says from 2000 to 2010. Journal of Interactive Learning Research, 25 (2), 261–280.

Robertson, B., Schumacher, L., Gosman, G., Kanfer, R., Kelley, M., & DeVita, M. (2009). Simulation-based crisis team training for multidisciplinary obstetric providers. Simululation in Healthcare, 4 (2), 77–83. doi: 10.1097/SIH.0b013e3181917cd .

Rutten, N., van Joolingen, W. R., & van der Veen, J. T. (2012). The learning effects of computer simulations in science education. Computers & Education, 58 (1), 136–153.

Salen, K., & Zimmerman, E. (2004). Rules of play: Game design fundamentals . Cambridge, MA, USA: MIT Press.

Sarabia-Cobo, C. M., Alconero-Camarero, A. R., Lavín-Alconero, L., & Ibáñez-Rementería, I. (2016). Assessment of a learning intervention in palliative care based on clinical simulations for nursing students. Nurse Education Today, 45 , 219–224.

Sauve, L., Renaud, L., Kaufman, D., & Marquis, J. S. (2007). Distinguishing between games and simulation: A systematic review. Education Technology & Society, 10 (3), 247–256.

Sawyer, B. (2002). Serious games: Improving public policy through game-based learning and simulation . USA: Woodrow Wilson International Center for Scholars.

Seng, W. Y., & Yatim, M. H. M. (2014). Computer game as learning and teaching tool for object oriented programming in higher education institution. Procedia-Social and Behavioral Sciences, 123 , 215–224.

Shieh, R. S., Chang, W., & Tang, J. (2010). The impact of implementing technology-enabled active learning (TEAL) in university physics in Taiwan. Asia-Pacific Education Researcher (De La Salle University Manila), 19 (3).

Shin, S., Park, J. H., & Kim, J. H. (2015). Effectiveness of patient simulation in nursing education: Meta-analysis. Nurse Education Today, 35 (1), 176–182.

Siewiorek, A., Saarinen, E., Lainema, T., & Lehtinen, E. (2012). Learning leadership skills in a simulated business environment. Computers & Education, 58 , 121–135.

Siewiorek, A., Gegenfurtner, A., Lainema, T., Saarinen, E., & Lehtinen, E. (2013). The effects of computer-simulation game training on participants’ opinions on leadership styles. British Journal of Educational Technology, 44 (6), 1012–1035.

Silvia, C. (2012). The impact of simulations on higher-level learning. Journal of Public Affairs Education, 18 (2), 397–422.

Sitzmann, T. (2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology, 64 (2), 489–528.

Smetana, L. K., & Bell, R. L. (2012). Computer simulations to support science instruction and learning: A critical review of the literature. International Journal of Science Education, 34 (9), 1337–1370.

Söderström, T., Häll, L., Nilsson, T., & Ahlqvist, J. (2014). Computer simulation training in health care education fuelling reflection-in-action? Simulation & Gaming, 45 (6), 805–828.

Stanley, D., & Latimer, K. (2011). ‘the Ward’: A simulation game for nursing students. Nurse Education in Practice, 11 (1), 20–25.

Strycker, J. (2016). Utilizing a simulation within an online school technology leadership course. Online Learning Journal, 20 (1), 130–144.

Swanson, E. A., Nicholson, A. C., Boese, T. A., Cram, E., Stineman, A. M., & Tew, K. (2011). Comparison of selected teaching strategies incorporating simulation and student outcomes. Clinical Simulation in Nursing, 7 (3), e81–e90.

Tanner, J. R., Stewart, G., Totaro, M. W., & Hargrave, M. (2012). Business simulation games: Effective teaching tools or window dressing? American Journal of Business Education (Online), 5 (2), 115.

Tao, Y. H., Yeh, C. C. R., & Hung, K. C. (2015). Validating the learning cycle models of business simulation games via student perceived gains in skills and knowledge. Educational Technology & Society, 18 (1), 77–90.

Terzidou, T., Tsiatsos, T., Dae, A., Samaras, O., & Chasanidou, A. (2012). Utilizing virtual worlds for game based learning: Grafica, a 3D educational game in second life. In 2012 IEEE 12th International Conference on Advanced Learning Technologies (pp. 624-628). IEEE.

Tiwari, S. R., Nafees, L., & Krishnan, O. (2014). Simulation as a pedagogical tool: Measurement of impact on perceived effective learning. The International Journal of Management Education, 12 (3), 260–270.

Tsekleves, E., Cosmas, J., & Aggoun, A. (2014). Benefits, barriers and guideline recommendations for the implementation of serious games in education for stakeholders and policymakers. British Journal of Educational Technology, 47 (1), 164–183. doi: 10.1111/bjet.12223 .

van Roessel, L., & van Mastrigt-Ide J. (2011). Collaboration and team composition in applied game creation processes. DiGRA '11, proceedings of the 2011 DiGRA international conference, think design play, 1-14.

von Wangenheim, C. G., Savi, R., & Borgatto, A. F. (2012). DELIVER!–an educational game for teaching earned value management in computing courses. Information and Software Technology, 54 (3), 286–298.

Vos, L., & Brennan, R. (2010). Marketing simulation games: Student and lecturer perspectives. Marketing Intelligence & Planning, 28 (7), 882–897.

Wang, C., Huang, C. C., Lin, S. J., & Chen, J. W. (2016). Using multimedia tools and high-fidelity simulations to improve medical students' resuscitation performance: An observational study. BMJ Open, 6 (9), e012195.

Wang, L. C., & Chen, M. P. (2010). The effects of game strategy and preference-matching on flow experience and programming performance in game-based learning. Innovations in Education and Teaching International, 47 (1), 39–52.

Warren, J. N., Luctkar-Flude, M., Godfrey, C., & Lukewich, J. (2016). A systematic review of the effectiveness of simulation-based education on satisfaction and learning outcomes in nurse practitioner programs. Nurse Education Today, 46 , 99–108.

White, B., Kahriman, A., Luberice, L., & Idleh, F. (2010). Evaluation of software for introducing protein structure. Biochemistry and Molecular Biology Education, 38 (5), 284–289.

Willoughby, T. (2008). A short-term longitudinal study of internet and computer game use by adolescent boys and girls: Prevalence, frequency of use, and psychosocial predictors. Developmental Psychology, 44 (1), 195–204.

Woo, J. C. (2014). Digital game-based learning supports student motivation, cognitive success, and performance outcomes. Educational Technology & Society, 17 (3), 291–307.

Wouters, P., & van Oostendorp, H. (2013). A meta-analytic review of the role of instructional support in game-based learning. Computers & Education, 60 (1), 412–425.

Wouters, P., Van Nimwegen, C., Van Oostendorp, H., & Van Der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105 (2), 249–265.

Yang, J. C., Chen, C. H., & Jeng, M. C. (2010). Integrating video-capture virtual reality technology into a physically interactive learning environment for English learning. Computers & Education, 55 (3), 1346–1356.

Yang, J. C., Chien, K. H., & Liu, T. C. (2012). A digital game-based learning system energy for energy education: An energy conservation pet. The Turkish Online Journal of Educational Technology, 11 (2), 27–37.

Yang, Y. C., & Chang, C. L. (2013). Empowering students through digital game authorship: Enhancing concentration, critical thinking, and academic achievement. Computers & Education, 68 , 334–344.

Yin, C., Song, Y., Tabata, Y., Ogata, H., & Hwang, G. J. (2013). Developing and implementing a framework of participatory simulation for mobile learning using scaffolding. Educational Technology & Society, 16 (2), 137–150.

Young, M. F., Slota, S., Cutter, A. B., Jalette, G., Mullin, G., Lai, B., Simeoni, Z., Tran, M., & Yukhymenko, M. (2012). Our princess is in another castle a review of trends in serious gaming for education. Review of Educational Research, 82 (1), 61–89.

Zacharia, Z. C., & Olympiou, G. (2011). Physical versus virtual manipulative experimentation in physics learning. Learning and Instruction, 21 (3), 317–331.

Zyda, M. (2005). From visual simulation to virtual reality to games. Computer, 38 (9), 25–32.

Download references

Acknowledgements

The research was sponsored by Laureate International Universities, through the “David Wilson Award for Excellence in Teaching and Learning”, won by Dr. Dimitrios Vlachopoulos (2015-2017).

Authors' contributions

DV conceived the study. AM conducted the literature review and prepared the summaries and critical reflection on the corresponding literature. DV participated in the design of the study and analysis. AM participated in the preparation of the article's structure, graphs, and reference list. Both authors read and approved the final manuscript.

Competing interest

The authors declared that they have no competing interest.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and affiliations.

Laureate Online Education / European University Cyprus, Amsterdam, Netherlands

Dimitrios Vlachopoulos

European University Cyprus, Nicosia, Cyprus

Agoritsa Makri

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Dimitrios Vlachopoulos .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Vlachopoulos, D., Makri, A. The effect of games and simulations on higher education: a systematic literature review. Int J Educ Technol High Educ 14 , 22 (2017). https://doi.org/10.1186/s41239-017-0062-1

Download citation

Received : 28 November 2016

Accepted : 30 May 2017

Published : 10 July 2017

DOI : https://doi.org/10.1186/s41239-017-0062-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Game-based learning
• Digital games
• Pedagogical use
• Higher education
• Learning outcomes
• Cognitive goals
• Behavioural goals
• Affective goals

• Help & FAQ

Simulation for manufacturing system design and operation: Literature review and analysis

• Engineering Division (Great Valley)

Research output : Contribution to journal › Review article › peer-review

This paper provides a comprehensive review of discrete event simulation publications published between 2002 and 2013 with a particular focus on applications in manufacturing. The literature is classified into three general classes of manufacturing system design, manufacturing system operation, and simulation language/package development. The paper further categorizes the literature into 11 subclasses based on the application area. The current review contributes to the literature in three significant ways: (1) it provides a wide coverage by reviewing 290 papers; (2) it provides a detailed analysis of different aspects of the literature to identify research trends through innovative data mining approaches as well as insights derived from the review process; and (3) it updates and extends the existing classification schemes through identification and inclusion of recently emerged application areas and exclusion of obsolete categories. The results of the literature analysis are then used to make suggestions for future research.

All Science Journal Classification (ASJC) codes

• Control and Systems Engineering
• Hardware and Architecture
• Industrial and Manufacturing Engineering

Access to Document

• 10.1016/j.jmsy.2013.12.007

Other files and links

• Link to publication in Scopus
• Link to the citations in Scopus

Fingerprint

• Systems analysis Engineering & Materials Science 100%
• Computer simulation languages Engineering & Materials Science 60%
• Discrete event simulation Engineering & Materials Science 47%
• Data mining Engineering & Materials Science 36%

T1 - Simulation for manufacturing system design and operation

T2 - Literature review and analysis

AU - Negahban, Ashkan

AU - Smith, Jeffrey S.

N1 - Copyright: Copyright 2014 Elsevier B.V., All rights reserved.

PY - 2014/4

Y1 - 2014/4

N2 - This paper provides a comprehensive review of discrete event simulation publications published between 2002 and 2013 with a particular focus on applications in manufacturing. The literature is classified into three general classes of manufacturing system design, manufacturing system operation, and simulation language/package development. The paper further categorizes the literature into 11 subclasses based on the application area. The current review contributes to the literature in three significant ways: (1) it provides a wide coverage by reviewing 290 papers; (2) it provides a detailed analysis of different aspects of the literature to identify research trends through innovative data mining approaches as well as insights derived from the review process; and (3) it updates and extends the existing classification schemes through identification and inclusion of recently emerged application areas and exclusion of obsolete categories. The results of the literature analysis are then used to make suggestions for future research.

AB - This paper provides a comprehensive review of discrete event simulation publications published between 2002 and 2013 with a particular focus on applications in manufacturing. The literature is classified into three general classes of manufacturing system design, manufacturing system operation, and simulation language/package development. The paper further categorizes the literature into 11 subclasses based on the application area. The current review contributes to the literature in three significant ways: (1) it provides a wide coverage by reviewing 290 papers; (2) it provides a detailed analysis of different aspects of the literature to identify research trends through innovative data mining approaches as well as insights derived from the review process; and (3) it updates and extends the existing classification schemes through identification and inclusion of recently emerged application areas and exclusion of obsolete categories. The results of the literature analysis are then used to make suggestions for future research.

UR - http://www.scopus.com/inward/record.url?scp=84897115618&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84897115618&partnerID=8YFLogxK

U2 - 10.1016/j.jmsy.2013.12.007

DO - 10.1016/j.jmsy.2013.12.007

M3 - Review article

AN - SCOPUS:84897115618

SN - 0278-6125

JO - Journal of Manufacturing Systems

JF - Journal of Manufacturing Systems