literature review on software development life cycle

Accessibility Links

• Skip to content
• Skip to search IOPscience
• Skip to Journals list
• Accessibility help
• Accessibility Help

Click here to close this panel.

Purpose-led Publishing is a coalition of three not-for-profit publishers in the field of physical sciences: AIP Publishing, the American Physical Society and IOP Publishing.

Together, as publishers that will always put purpose above profit, we have defined a set of industry standards that underpin high-quality, ethical scholarly communications.

We are proudly declaring that science is our only shareholder.

Software Development Life Cycle early phases and quality metrics: A Systematic Literature Review

Shokhista Ergasheva 1 and Artem Kruglov 1

Published under licence by IOP Publishing Ltd Journal of Physics: Conference Series , Volume 1694 , Information Technology, Telecommunications and Control Systems (ITTCS), 2020 17-18 December 2020 Innopolis, Russia Citation Shokhista Ergasheva and Artem Kruglov 2020 J. Phys.: Conf. Ser. 1694 012007 DOI 10.1088/1742-6596/1694/1/012007

Article metrics

6355 Total downloads

Share this article

Author e-mails.

[email protected]

Author affiliations

1 Russia, Innopolis University

Buy this article in print

Most of the currently used software development metrics are concentrated on the latter stages like development and testing. However, early revealing of errors during the SDLC(Software Development Life Cycle) tremendously affects the efficiency of the team work by spending more time on prevention and less on correction in later stages.

Furthermore, reworking in later stages increase the cost of quality, lead to extra waste of time of the development team. The objective of this review is to examine the classification of the existing SDLC(Software Development Life Cycle) early phases and define the set of software process quality metrics. Based on the SRL research protocol, we selected the most relevant studies from overall 200 publications by the use of search keywords and inclusion/exclusion criteria for quality assessment of primary studies. This systematic literature review yields the correlation of cost, time and software product quality with the SDLC stages.

Export citation and abstract BibTeX RIS

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence . Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

Review of System Development Life Cycle (SDLC) Models for Effective Application Delivery

• Conference paper
• First Online: 27 July 2021
• Cite this conference paper

• Oluwaseyi Ezekiel Olorunshola 13 &
• Francisca Nonyelum Ogwueleka 13  

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 191))

1337 Accesses

8 Citations

There are different system development models, tools, and applications that have been designed and developed before the use of computers for processing of information, and there is an increase in demand of software with cheaper cost, having more functionality, faster delivery, and of high quality than how it was previously. Therefore, there is need to know the methods and their applications that would conform to the organizational requirements for successful system deployment. Each method has constructive criticisms with various advantages and disadvantages to the system that are of importance for deploying software in a manner such that it helps in good decision making on a chosen method for delivery within deadline and proper quality. This paper explained four commonly used system development methods namely; waterfall, iterative, agile, and rapid application development (RAD). It thereafter categorically assesses these methods on a comprehensive set of features and made an alternative analysis of the applicability of the methods based on system development life cycle tools which are requirement, design, implementation, and testing. This paper describes the system development life cycle (SDLC) tools and applications for successful system deployment and the constructive comparison that should serve as a tool in model selection for system development.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Balaji, S., Murugaiyan, M.S.: Waterfall vs. V-model vs. agile: a comparative study on SDLC. Int. J. Inf. Technol. Bus. Manag. 2 (1), 26–30 (2012)

Google Scholar  

Berger, H., Beynon-Davies, P.: The utility of rapid application development in large‐scale, complex projects. Inf. Syst. J. 19 (6), 549–570 (2009)

Berra, Y.: Software Development Life Cycle (SDLC). DePaul Univ. 2 12, 2012. https://condor.depaul.edu/jpetlick/extra/394/Session2.ppt (2012)

Dennis, A., Wixom, B.H., Roth, R.M.: Systems Analysis and Design. Wiley & Sons (2018)

Highsmith, J., Cockburn, A.: Agile software development: the business of innovation. Computer (Long. Beach. Calif) 34 (9), 120–127 (2001)

Kosta, E., Pitkänen, O., Niemelä, M., Kaasinen, E.: Mobile-centric ambient intelligence in health-and homecare—anticipating ethical and legal challenges. Sci. Eng. Ethics 16 (2), 303–323 (2010)

Article   Google Scholar  

Land, S.K., Smith, D.B., Walz, J.W.: Practical Support for Lean Six Sigma Software Process Definition: Using IEEE Software Engineering Standards, vol. 70. Wiley (2012)

Petersen, K., Wohlin, C., Baca, D.: The waterfall model in large-scale development. In: International Conference on Product-Focused Software Process Improvement, pp. 386–400 (2009)

Tiruneh, T., Mishra, M.K.: Analysis and performance evaluation of requirement elicitation techniques. Int. J. Comput. Sci. Eng. 6 (4), 118–123 (2018)

Tiwari, S., Rathore, S.S.: A methodology for the selection of requirement elicitation techniques 3 (2), 212–217 (2017). arXiv Prepr. arXiv1709.08481

Tousif, R., Khan, M.N.A., Riaz, N.: Analysis of requirement engineering processes, tools/techniques and methodologies. Int. J. Inf. Technol. Comput. Sci. 5 (3), 40–48 (2013)

Unhelkar, B.: The Art of Agile Practice: A Composite Approach for Projects and Organizations. Auerbach Publications (2016)

Yousuf, M., Asger, M.: Comparison of various requirements elicitation techniques. Int. J. Comput. Appl. 116 (4) (2015)

Download references

Author information

Authors and affiliations.

Computer Science Department, Nigerian Defence Academy, Kaduna, Nigeria

Oluwaseyi Ezekiel Olorunshola & Francisca Nonyelum Ogwueleka

You can also search for this author in PubMed   Google Scholar

Editor information

Editors and affiliations.

Global Knowledge Research Foundation, Ahmedabad, Gujarat, India

Computing and Technology, Nottingham Trent University, Nottingham, Nottinghamshire, UK

Mufti Mahmud

University of Peradeniya, Kandy, Sri Lanka

Roshan G. Ragel

Prof Ram Meghe College of Engineering and Management, Amravati, India

Nileshsingh V. Thakur

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Cite this paper.

Olorunshola, O.E., Ogwueleka, F.N. (2022). Review of System Development Life Cycle (SDLC) Models for Effective Application Delivery. In: Joshi, A., Mahmud, M., Ragel, R.G., Thakur, N.V. (eds) Information and Communication Technology for Competitive Strategies (ICTCS 2020). Lecture Notes in Networks and Systems, vol 191. Springer, Singapore. https://doi.org/10.1007/978-981-16-0739-4_28

Download citation

DOI : https://doi.org/10.1007/978-981-16-0739-4_28

Published : 27 July 2021

Publisher Name : Springer, Singapore

Print ISBN : 978-981-16-0738-7

Online ISBN : 978-981-16-0739-4

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Software Development Life Cycle early phases and quality metrics: A Systematic Literature Review

• Ergasheva, Shokhista
• Kruglov, Artem

Most of the currently used software development metrics are concentrated on the latter stages like development and testing. However, early revealing of errors during the SDLC(Software Development Life Cycle) tremendously affects the efficiency of the team work by spending more time on prevention and less on correction in later stages. Furthermore, reworking in later stages increase the cost of quality, lead to extra waste of time of the development team. The objective of this review is to examine the classification of the existing SDLC(Software Development Life Cycle) early phases and define the set of software process quality metrics. Based on the SRL research protocol, we selected the most relevant studies from overall 200 publications by the use of search keywords and inclusion/exclusion criteria for quality assessment of primary studies. This systematic literature review yields the correlation of cost, time and software product quality with the SDLC stages.

Accessibility in the Software Development Life Cycle: A Systematic Literature Review

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Open access
• Published: 09 November 2016

Game development software engineering process life cycle: a systematic review

• Saiqa Aleem   ORCID: orcid.org/0000-0002-3385-0613 1 ,
• Luiz Fernando Capretz 2 &
• Faheem Ahmed 3  

Journal of Software Engineering Research and Development volume  4 , Article number:  6 ( 2016 ) Cite this article

93k Accesses

62 Citations

5 Altmetric

Metrics details

Software game is a kind of application that is used not only for entertainment, but also for serious purposes that can be applicable to different domains such as education, business, and health care. Multidisciplinary nature of the game development processes that combine sound, art, control systems, artificial intelligence (AI), and human factors, makes the software game development practice different from traditional software development. However, the underline software engineering techniques help game development to achieve maintainability, flexibility, lower effort and cost, and better design. The purpose of this study is to assesses the state of the art research on the game development software engineering process and highlight areas that need further consideration by researchers. In the study, we used a systematic literature review methodology based on well-known digital libraries. The largest number of studies have been reported in the production phase of the game development software engineering process life cycle, followed by the pre-production phase. By contrast, the post-production phase has received much less research activity than the pre-production and production phases. The results of this study suggest that the game development software engineering process has many aspects that need further attention from researchers; that especially includes the postproduction phase.

1 Introduction

With the rapid advancement of computer technology, the significance of software engineering in our daily lives is increasing. It affects every aspect of our lives today, including working, living, learning, and education. A new and popular mode of entertainment and an important application of technology are software games, which have become increasingly accepted by people of all ages. In today’s culture, technology is easily accessible and has become more convenient; more and more people like to play games and are also becoming motivated to design their own games. Salen and Zimmerman ( 2003 ) defined “game is a software application in which one or more players make decisions by controlling game objects and resources, in the pursuit of its goal”. Software games are software applications that are installed on hardware devices such as video game consoles, computers, handheld devices, and Personal Digital Assistants (PDAs). Software games have now become a worldwide creative industry, but because of the multidisciplinary activities required, their development is a very complex task.

The multidisciplinary nature of the processes that combine sound, art, control systems, artificial intelligence (AI), and human factors, also makes the software game development practice different from traditional software development. However, despite the high complexity of the software engineering development process, the game industry is making billions of dollars in profit and creating many hours of fun (PWC, 2011–2014 outlook). The software game market throughout the world has grown by over 7–8 % annually and has reached sales of around $5.5 billion in 2015 (SUPERDATA 2015 ). Newzoo Game Market ( 2015 ) has also reported that the world-wide digital game market will reach $113.3 billion by 2018.

Creation of any game involves cross-functional teams including designers, software developers, musicians, script writers, and many others. Also, Entertainment Software Association (ESA) ( 2014 ); 2015 ) reports highlighted the latest trends about the software game industry. Therefore, game development careers have currently become highly challenging, dynamic, creative, and profitable (Liming and Vilorio, 2011 ). The ability to handle complex development tasks and achieve profitability does not happen by chance, but rather a common set of good practices must be adopted to achieve these goals. The game industry can follow the good and proven practices of traditional software engineering, but only a clear understanding of these practices can enhance the complex game development engineering process.

The computer game domain covers a great variety of player modes and genres (Gredler, 1995 ; Gredler, 2003 ; Rieber, 2005 ). The complexity of software games has posed many challenges and issues in software development engineering process because it involves diverse activities in creative arts disciplines (storyboarding, design, refinement of animations, artificial intelligence, video production, scenarios, sounds, marketing, and, finally, sales) in addition to technological and functional requirements (Keith, 2010 ). This inherent diversity leads to a greatly fragmented domain from the perspectives of both underlying theory and design methodology. The software game literature published in recent years has focused mainly on technical issues. Issues of game production, development, and testing reflect only the general software-engineering state of the art. Pressman ( 2001 ) states that a game is a kind of software that entertains its users, but game development software engineering faces many challenges and issues if only a traditional software-development process is followed (Kanode and Haddad, 2009 ; Petrillo et al., 2009 ). Some studies have proposed a Game Development Software Engineering (GDSE) process life cycle that provides guidelines for the game development software engineering process (Hendrick, 2014 ; Blitz game studio, 2014 ; McGrath, 2014 ; Chandler, 2010 ; Ramadan and Widyani 2013 ). However, the proposed GDSE process life cycle development phases do not ensure a quality development process.

A GDSE process is different from a traditional software development engineering process, and all phases of the proposed GDSE process life cycle can be combined into three main phases: pre-production, production, and post-production. The pre-production phase includes testing the feasibility of target game scenarios, including requirements engineering marketing strategies; the production phase involves planning, documentation, and game implementation scenarios with sound and graphics. The last phase post-production involves testing, marketing, and game advertising. Because of high competition and extreme market demand, game development companies sometimes reduce their development process so they can be first to market (Kaitilla, 2014 ). This reduction of the development process definitely affects game quality. Because of these types of complex project-management tasks, the game development software engineering process diverges from traditional software development. Therefore, it becomes important now to investigate the challenges or issues faced by game development organizations in developing good quality games. This systematic literature review is the first step towards identifying the research gaps in the GDSE field.

1.1 Related work

Managing GDSE process life cycle has become a much harder process than anyone could have initially imagined, and because of the fragmented domain, no clear picture of its advancement can be found in the literature. A systematic literature review provides a state of the art examination of an area and raises open research questions in a field, thus saving a great deal of time for those starting research in the field. However, to the best of the authors’ knowledge, no systematic literature review has been reported for GDSE process life cycle. Many researchers have adopted the systematic literature review approach to explore different aspects in software games. Boyle et al. ( 2012 ) conducted a systematic literature review to explore the engagement factor in entertainment games from a player’s perspective. In this study, 55 papers were selected to perform the systematic literature review. The study highlighted the different aspects of engagement factors with entertainment games; these include subjective feelings of enjoyment, physiological responses, motives, game usage, player loyalty, and the impact of playing games on a player’s life. Connolly et al. ( 2012 ) explored 129 papers to report the impacts and outcomes of computer and serious games with respect to engagement and learning by using the systematic literature review approach.

Another study also reported the importance of engagement in digital games by using a systematic literature review approach. Osborne-O’Hagan et al. ( 2014 ) performed a systematic literature review on software development processes for games. A total of 404 studies were analyzed from industry and academia and different software development adoption models used for game development were discussed. The findings of the study were that qualitative studies reported more agile practices than the hybrid approach. The quantitative studies used an almost hybrid approach. We also noted that lightweight agile practices such as Scrum, XP, and Kanban – are suitable where innovation and time to market is important. A risk-driven spiral approach is appropriate for large projects. Only one systematic study was performed related to research on software engineering practices in the computer game domain rather than GDSE process life cycle (Ampatzoglou and Stamelos 2010 ).

This study mainly review the existing evidence in the literature concerning the GDSE process research and suggest areas for further investigation by identifying possible gaps in current research. Furthermore, the aim of this study is to cover the state of the art for the GDSE process life cycle, and to accomplish this, an evidence-based research paradigm has been used. In the software engineering field, possible use of an evidence-based paradigm have been proposed by Dyba et al. ( 2005 ) and Kitchenham et al. (2004). The Systematic Literature Review (SLR) research paradigm constitutes the first step in an evidence-based paradigm research process, and its guidelines for performing systematic research are thoroughly described by Brereton et al. ( 2007 ) and Kitchenham ( 2004 ).

The rest of the paper is organized as follows: Section 2 provides the research background and Section 3 describes the methodology used for the systematic literature review as described by Breton et al. (2007). Section 4 presents the statistics for the primary studies, Section 5 answers various research questions, Section 6 discuss the external threats to validity, and, finally, Section 7 concludes the presentation.

2 Background

In the software development industry, software games are gaining importance because they are not only used for entertainment, but also for serious purposes that can be applicable to different domains such as education, business, and health care. Serious games are designed to have an impact on the target audience similar to entertainment games but they are combined seemingly with a practical dimension too. Both have to be attractive and appealing to a broad target audience (Alvarez & Michaud, 2008 ). Especially for serious games, along with their applicability to different domains, their revenue has also been increasing. Games software earned three times more revenue than any other software product in 2012 (Nayak, 2013 ).

Robin ( 2009 ) defines a development method as a systematized procedure to achieve the goal of producing a working product within budget and on schedule. A number of methodologies used for game development and design (Castillo 2008 ). The first is the waterfall method, which is also commonly used in traditional software development. Unlike game projects, once the pre-production phase is completed, production phase activities are performed in a “waterfall” manner. First, the activities are segregated based on functionalities and assets, and then they are assigned to their respective teams. The requirements team spent a significant amount of time in functionality definition and front-end activities, which implies a late implementation of level and mechanisms (Schwaber & Beedle, 2002 ). However, in the waterfall method, it is difficult to reverse any activity (Flood, 2003 ).

The second development methodology is the agile method that is commonly used for game development. These methods are highly iterative and not documentation-centric. The production phase is divided into small iterations and focusses on the most crucial features. During the beginning phase of each iteration, the whole team meets and sets clear objectives. At the end of each iteration, results are communicated to clients. These methods support different team cycles and dynamics through daily meetings. The most used agile methodologies in game development are extreme programming (XP), rapid prototyping, and Scrum (Godoy & Barbosa, 2010 ).

The unified development process (Kruchten, 2000 ) is another traditional SE method, which focusses more on analyzing requirements and converting them into functional software components. The requirement analysis document includes a definition of the game concept, use cases, and assets definitions (Schwaber & Beedle, 2002 ). The method includes five disciplines: requirements, analysis, design, implementation, and testing. The unified process is based on a philosophy of four key elements: iterative and incremental, use case-driven, architecture-centric, and risk-driven.

Kanode and Haddad ( 2009 ) stated that an important, but incorrect, assumption was made that GDSE follows the waterfall method. More recently, researchers have agreed that it must follow the incremental model (Munassar and Govardhan 2010 ) because it combines the waterfall method with an iterative process. A major concern, reported by Petrillo et al. ( 2009 ), was that very poor development methodologies are commonly used by developers for software creation in the game industry. The GDSE appears as a question in many forms attempting to determine what types of practices are used. However, there is no single answer to this question. Few researchers have explored GDSE practices and then tried to answer questions like the phases of the GDSE process life cycle. Blitz game studios ( 2014 ) proposed six phases for the GDSE process life cycle: Pitch (initial design and game concept), Pre-production (game design document), Main production (implementation of game concepts), Alpha (internal testers), Beta (third-party testers), and the Master phase (game launch). Hendrick ( 2014 ) proposed a five-phase GDSE process life cycle consisting of Prototype (initial design prototype), Pre-production (design document), Production (asset creation, source code, integration aspects), Beta (user feedback), and, finally, the Live phase (ready to play). McGrath ( 2014 ) divided the GDSE process life cycle into six phases: Design (initial design and game design document), Develop/redevelop (game engine development), Evaluate (if not passed, then redevelop), Test (internal testing), Review release (third-party testing), and Release (game launch). Another GDSE process life cycle proposed by Chandler ( 2010 ) consisted of four phases: Pre-production (design document and project planning), Production (technical and artistic), Testing (bug fixing), and, finally, the Post-production phase (post-mortem activities). The latest GDSE process life cycle in 2013 proposed by Ramadan and Widyani ( 2013 ) was based on the four GDSE process life cycles previously described. They proposed six phases: Initiation (rough concept), Pre-production (creation of game design and prototype), Production (formal details, refinement, implementation), Testing (bug reports, refinement testing, change requests), Beta (third-party testers), and Release (public release).

In traditional software engineering, the development phase usually involves activities such as application design and its implementation; the production phase is when the software actually runs and is ready for use. However, in the GDSE process lifecycle, the production phase includes the development process, which is the pre-production phase of the traditional software engineering process, and the production phase of traditional software engineering is actually the post-production phase of the GDSE process life cycle (Bethke, 2003 ). Therefore, the GDSE process life cycle is different from the traditional software engineering process, and many researchers have studied the challenges faced by this domain (Kanode and Haddad, 2009 ). The most prominent observation made in these studies is that to address the challenges faced by the GDSE process life cycle, more rigorous software engineering strategies must be used. Most researchers have explicitly compared the software engineering process with the GDSE process, but none of them has studied complete GDSE process life cycle and research topics under this domain in detail. This study will provide evidence on these topics and their differences from the traditional software engineering process. In this paper, the GDSE process phases were divided into three phases for basic understanding: Preproduction, Production, and Post-production. Efforts were made to classify these further based on studies found in the literature. The primary contribution of this paper is that it is the first SLR that addresses these GDSE process life cycle research topics and highlights the topics that need further attention by researchers.

In this work, the conceptual description of the SLR process presented by Kitchenham ( 2004 ) was used to investigate the research intensity for each phase of the GDSE process life cycle. Conceptually, SLR provides an opportunity for researchers to collect empirical evidence from the existing literature about a formulated research question. Although most authors followed the general SLR guidelines provided by Kitchenham ( 2004 ), there were slight variations in the description and presentation of the conceptual process layout. The generic SLR guidelines stated by Kitchenham ( 2004 ) are further elaborated here, and the overall process is described as a set of activities The research process has been adopted for this study described by Kitchenham and Charters ( 2007 ). There are mainly three phases of the review and the steps associated with each phase are shown in Fig.  1 .

3.1 Planning phase (Step 1–4)

This study started by selecting a topic, at which point the study objectives were also clearly defined and the boundaries of the domain delineated.

3.1.1 Selection of topic and research questions

Selecting a topic for SLR is of crucial importance because many factors such as individual or community interest, research gaps, and research impact contribute to shaping research questions on the topic. Our understanding of the GDSE process life cycle is continuously evolving (Kitchenham et al., 2010 ), and many areas in this field lack generalized evidence. It is critically important for the game industry to identify a quality-driven GDSE process. Several studies have investigated different phases of the GDSE process life cycle, but they do not offer systematic, comprehensive, and thorough methodological research specific to this topic.

In this review, studies from 2000 to 2015 will be explored to answer the following research questions:

Research Question (RQ1): What is the intensity of research activity on the GDSE process life cycle?

RQ2: What topics are being researched in the pre-production, production, and post-production phases?

RQ3: What research approaches are being used by researchers in the software game domain?

RQ4: What empirical research methods are being used in the software game domain?

The number of publications has been identified by the research group to address RQ1. A graphical representation has been used to represent the increase or decrease in the number of publications per year as a measure of research activity. To address RQ2, RQ3, and RQ4, each study selected has been affiliated to a research topic, to a certain approach, and to a specific methodology used for the research. Details of this classification into corresponding categories are discussed in section 3.2.4 .

3.1.2 Review team & protocol establishment

A multidisciplinary team is needed to perform a high-quality scientific SLR. To enhance the thoroughness and minimize the potential bias of a study, an SLR is normally undertaken by more than one reviewer. The SLR team for this review was made up of three people. Two people were designated as principal reviewers (Second expert report by American institute 2011). One person was also selected as the project leader to handle additional administrative tasks such as team communication, points of contact, meeting arrangements and documentation, task assignment and follow-up, and quality assurance. Table  1 details the tasks required for the SLR process and reviewer’s involvement and total time duration.

In order to ensure the review could be replicated and to reduce researcher bias a review protocol and it’s evaluation procedure was developed at step 3 and 4. The final review protocol is discussed in the following sections 3.2.1 to 3.2.4 (Steps 5–9 incl.).

3.2 Conducting phase (Step 5–9)

3.2.1 search strategy.

In the SLR, the search procedure is based on an online search. The search strategy for an SLR is a plan to construct search terms by identifying populations, interventions, and outcomes. Key terms are combined together to created different groups in order to form search strings. Each group comprise of terms that are either different forms of the same word, synonyms, or terms that have similar or related semantic meaning within the domain. Table  2 depicts the followed approach.

In order to retrieve different sets of relevant literature, four groups are designed. The main objective of this grouping is to find the literature that is the intersection of the groups as shown in Fig.  2 .

Selection of relevant studies

The search strategy was implemented by applying the “AND” and “OR”, where the “OR” operator is used within the Group and the “AND” is used between the groups. According to Table  2 , the following search string will capture the structure:

( Group 1: [Software game] OR [Digital game] OR [Video game] OR [Computer game] OR [Online Game] OR [Serious games] OR [Educational Games] OR [Learning Games])

( Group 2: [Development] OR [Advancement] OR [Steps] OR [Evolve] OR [Project])

( Group 3: [Life cycle] OR [Design] OR [Implementation] OR [Requirements Engineering] OR [Testing] OR [Evaluation] OR [Maintenance])

( Group 4: [Process] OR [Progression] OR [Method] OR [Model]).

Therefore, “ Software game development lifecycle process ”, “ Computer game development design process ” and “ video game testing process” are some examples of the search strings and similar way different search strings were formed in order to capture all relevant studies.”

To ensure that all relevant research concerning this area of study was reviewed, journals and conferences from 2000 to 2015 were covered, using as sources IEEE Explorer, ACM Digital Library, Science Direct Elsevier, Taylor & Francis, Google Scholar, and Wiley Publications. If the information required, as indicated on the form shown in Table  3 , was not explicitly present in the potential study, then that paper was peer-reviewed by all team members and, after discussion, validated for correctness. Otherwise, each paper was reviewed by one reviewer. Each study involved some general information and some specific information, as indicated on the form.

3.2.2 Pilot selection & data extraction

The research study selection and data extraction was based on the following coverage criteria:

Inclusion criteria for study

For SLR, articles and research papers from 2000 to 2015 were included, and to evaluate their suitability, the following criteria were analyzed:

The study should be thoroughly reviewed by at least one of the reviewers.

Only the following types of studies were considered: case studies, theoretical papers, and empirical analysis surveys.

The full text of the article should be available.

If any article identifies any challenges and problems in software games, that article is included as a review.

Studies that describe motivation for game application.

Study exclusion criteria

The following criteria were used to determine articles to be excluded:

Articles published on company Web sites.

Articles not relevant to the research questions.

Articles not describing any phase of the game development life cycle.

Study selection

This procedure involved two phases. In the first phase, an initial selection was made on the basis of the inclusion criteria and after reading the title, abstract, and conclusion of each article. In the second phase, if a particular article met the criteria, then the whole article was studied. One hundred forty-eight papers were identified after final selection, as shown in Fig.  3 . Table  4 shows the results found in each data source and Additional file 1 : Appendix A contains a full list of selected publications.

Study selection process

3.2.3 Quality criteria

In this research, quality guidelines were defined based on a quality instrument that was used to assign a quality score to each article as a basis for data analysis and synthesis. The quality instrument consisted of four sections: a main section containing a generic checklist applicable to all studies, and three other sections specific to the type of study.

The checklist was based upon SLR guidelines (Kitchenham, 2004 ) and was derived from Kitchenham ( 2004 ) and Second expert report by American institute (2011). The detailed checklist is shown in Table  5 . Some of the checklist items could be answered by “yes” or “no” and they also included a “partial” option. A value of 1 was assigned to “yes,” 0 to “no,” and 0.5 to “partial”; then the sum of the checklist values was used to assign a quality score to the study to assess document quality.

3.2.4 Data synthesis

For data synthesis the topics, research approaches and methods are classified and their classification details are listed below:

Classification of topics in the GDSE Life Cycle

This section includes a classification of the topics covered by each study with respect to the pre-production, production, and post-production phase issues involved. The 2012 ACM classification system was used for classification, which is the same method used by Cai and Card ( 2008 ). The proposed classification system has been adopted by many journals and conferences specifically for software engineering topics. The same classification was used here to classify the papers under study, and these were further fabricated based on studies found in the GDLC domain. Table  6 presents the selected classification schema.

Research approaches and methods classification

Research articles can be characterized based on their method and approach, as described by Glass et al. ( 2002 ). The main categories for scientific approach are descriptive (a system, tool, or method; a literature review can also be considered as descriptive studies), exploratory (performed where a problem was not clearly defined), and empirical (findings based on observation of its subjects). To evaluate new methods or techniques, three major empirical research methods are used: surveys, case studies, and experiments (Wohlin et al., 2000 ). Table  7 describes the three major empirical research types; Dyba and Dingsoyr ( 2008 ) also used the same type of empirical classification.

The data collected were statistically analyzed as follows:

To address RQ1, the number of studies published per year, whether journal articles or conference publications, and the number of publications on the GDLC hosted by each digital library.

To address RQ2, the major topics of the GDLC that were investigated in the software game domain.

To address RQ3 and RQ4, the research approach or method used by number of studies.

From Section  3.2.4 , data were tabulated and are presented in Additional file 2 : Appendix B.

3.3 Documenting (Step 10–12)

This step of the SLR describe conclusion, possible threats and limitations to the validity of this study. Authors believe that there is a chance that the word game was not part of the title of some studies, but that nevertheless they discussed game development. These studies may, therefore, have been excluded from the primary dataset by the search procedure. There are other threats that are also linked to a systematic literature review such as generalization and subjective evaluation (Shadish et al., 2002 ).

There are limitations to our results, although significant amounts of effort and time was spent to select the papers that were studied. More specifically, our search was limited to the academic databases. It is obvious from the results of RQ1 that developers prefer to submit their work on the blogs or forums. However, posts for different game forums and blogs cannot be included in a systematic literature review because they don’t fulfil the quality criteria used for the selection of papers. In addition, the exclusion of less-known journals and conferences from the Web of Science and the Scopus index might have led to a different dataset.

Another limitation of the study is the exclusion of Human-Computer Interaction (HCI) filed studies. In the phase of screening out, we found studies from HCI field such as (Plass-Oude Boss et al. ( 2010 )) for games but they didn’t focus on software engineering perspective. In short, we didn’t consider studies from HCI because they take non-functional requirements, and usability features into account. These methods help developers to evaluate software and they considered as an integral part of game development. However, due to the limited scope of the study, we excluded studies from HCI field.

Finally, the classification scheme might have altered the results if they were classified by a scheme, such as the waterfall model, instead of the ACM classification scheme. Despite these limitations, the results of our systematic literature review will be useful to game development organizations and developers of software games.

4 Results and Discussion

This section presents the results of statistical analysis of the data set discusses the findings concerning the RQs formulated in Section 3.1 . The characteristics of the data set are tabulated for better understanding. To trace the categories of each mapped study, the interested reader is referred to the Additional file 2 : Appendix B. A total 148 studies were collated and analyzed as part of this review. To identify GDSE process life cycle domain specific characteristics, the findings of this review will be compared to results from similar studies done by Cai and Card ( 2008 ), Glass et al. ( 2002 ), and Dyba and Dingsoyr ( 2008 ).

4.1 RQ1 What is the intensity of research activity on the GDSE process life cycle?

Table  8 clearly shows that GDSE process life cycle research intensity has increased during the last few years. Figure  4 showed an increase in GDSE process life cycle over time. The y -axis represents the number of publications in the form of a fraction and is calculated by taking year (i) ’s number of publications as the numerator and year (0) ’s number of publications as the denominator. From Table  8 , 2007 was taken as year (0) , and the first data point of the graph was calculated for year (1) i.e., 2008. Figure  4 shows the results up to 2015. Years are given on the x- axis.

Increase in GDSE process life cycle research activity

Figure  4 illustrates that during the last few years, research activity in the GDSE process life cycle domain has continuously increased and the number of publications in the GDSE domain has increased at a polynomial growth rate since 2005. During 2013, 2014 and 2015 the drop in research activity is noted. It seems obvious that most of the work related to GDSE research activity was not published on the selected sources for this study. During 2014, most of the research activities were seen on the game development associations/groups web sites, like DIGRA association and Gamastura, or game developers personal blogs.

Moreover, Fig.  5 shows the list of countries most active in GDSE process life cycle topics research. Looking at research activity based on countries, China now dominates GDSE process life cycle research, but its research into the game domain started only in 2010. In four years, China has come to dominate this area of research. Before 2010, the United States and the United Kingdom were dominant.

Research activity per country

Authors from North and South America have played a dominant role since 2004 and are still contributing in this area. Contributors in Europe also started research into the GDSE domain in 2007, but the Asian continent has dominated the GDSE domain since 2010. It can be visualized in Fig.  6 . The most popular venue for GDSE research publication is IEEE; it seems that IEEE accounts for the main bulk of publications (approximately 63 %), followed by Elsevier, Springer, and ACM.

Research activity by continent

4.2 RQ 2: What topics are being researched in the pre-production. Production and post production phase?

This section addresses the identification of main research topics in the GDSE process life cycle domain. Table  9 clearly suggests that most research has been conducted in the production phase, followed by the pre-production phase. On the other hand, the post-production phase has not attracted much research interest. These GDSE process life cycle topics are somewhat different than in software engineering because of two factors: first, the GDSE domain has special needs and priorities, and second, it is a young domain which requires more fundamental research in the area of requirements, development, and coding tools. When the GDSE domain becomes mature, then other areas in the field, like testing and verification, will attract the interest of researchers.

As mentioned earlier in Section 2 , games have specific characteristics, which the conventional software development process cannot completely address. In the past years, research on GDSE process life cycle topics has become more active because, unlike other software products, games provide entertainment and user enjoyment, and developers need to give more importance to these aspects. As a result, research about the pre-production phase has increased. The implementation phase is shorter than in the traditional software implementation process because of the short time to market. This production-phase research intensity has attracted the interest of many researchers, and maximum research activity has been reported because the GDSE domain requires efficient development and coding techniques. McShaffry ( 2003 ) also highlighted the importance of the production phase to counteract poor internal quality. There is much less research activity in the post-production phase than in the pre-production and production phases.

Figure  7 presents the growth of each GDSE process life cycle research topic since 2000. It is apparent that in the pre-production phase, the most researched topic is management of the game development process, followed in this order by production-phase development platforms, programming, and implementation topics. In the post-production phase, the marketing area attracted the largest amount of research interest. The state of the art research is the description of actual primary studies, and, therefore, they are mapped according to the research topics they addressed (Budgen et al., 2008 ). Next, a short description of each GDSE topic is presented along with a full reference list. A full reference list of all the studies included is presented in Additional file 1 : Appendix A.

GDSE process life cycle research topics

4.2.1 Pre-production phase

In the pre-production phase, most of the studies categorized under this topic address management issues during the GDSE process life cycle. The overall management of the game development process combines both an engineering process and creation of artistic assets. Ramadan and Widyani [S1] compared various game development strategies from a management perspective, and most studies like [S3], [S6], [S7], and [S8] have proposed frameworks for game development. Game development guidelines can be followed to manage GDSE process life cycle. The presence of agile practices in the game development processes is also highlighted by some studies. Tschang [S4] and Petrillo et al. [S17] highlighted the issues in the game development process and their differences from traditional software development practices. Management of development-team members and their interaction is critically important in this aspect.

Some studies [S10] and [S11] have provided data analytics and empirical analysis of the game development process and issues of interdisciplinary team involvement. Best management practices in the game development process must consider certain elements such as staying on budget, timing, and producing the desired output. To assess game quality, five usability and quality criteria (functional, internally complete, balanced, fun, and accessible) can be used, but a process maturity model specific to the game development process is still needed to measure these processes for better management and high performance.

Requirements specification

One of the main differences between the traditional software development process and GDSE process life cycle is the requirements phase. The game development process requires consideration of many factors such as emotion, game play, aesthetics, and immersive factors. In four studies, the authors have discussed the requirements engineering perspective to highlight its importance for the whole game-software development process. They discussed emotional factors, language ontology, elicitation, feedback, and emergence [S19], [S20], [S21], and [S22]. In particular, game developers must understand these basic non-functional requirements along with the game play requirements and incorporate them while developing games. The main challenges in requirements identification are a) communication between diverse background stakeholders, b) non-functional requirements incorporation with game play requirements, such as media and technology integration, and c) validation of non-functional requirement such as fun, which is very complex because it is totally dependent on the target audience. Callele et al. [S20] further fabricated a set of requirements based on emotional criteria, game-playing criteria (cognitive factors and mechanics), and sensory requirements (visual, auditory, and haptic). The requirements specification phase must address both the functional and non-functional requirements of game development.

Game system description language

Many description languages are currently used by developers, such as the UML model, agent-based methodologies, and soft-system methodologies. Quanyin et al. [S32] proposed the UML model for mobile games. They performed experiments and reported that it would be a good model for further development of games on the Android operating system. Shaker et al. [S33] extracted features of the Super Mario Brothers game from different levels, frequency sequences of level elements, and statistical design levels. Then, they analyzed the relationship between a player’s experience and the level design parameters of platform games using feature analysis modelling. Tylor et al. [S28] proposed a soft system methodology for initial identification of game concepts in the development process. The proposed approach can be used instead of a popular description language because it provides an overview of the game. Chan and Yuen [S30] and Rodriguez et al. [S31] proposed an ontology knowledge framework for digital game development and serious games modelling using the AOSE methodology. A system description language for games must be both intelligible to human beings and formal enough to support comparison and analysis of players and system behaviors. In addition, it must be production-independent, adequately describe the overall game process, and provide clear guidelines for developers.

Reusability

The existence of reusability of software (Capretz and Lee 1992 ) and development platforms in game development has been reported by some researchers, but to gain its full advantages, commonality and variability analysis must be done in the pre-production phase. This category addresses reuse techniques for game development software (Ahmed and Capretz, 2011 ). Neto et al. [S34] performed a survey that analyzed game development software reuse techniques and their similarity to software product lines. Reuse techniques in game development could reduce cost and time and improve quality and productivity. For reuse techniques, commonality and variability analysis is very important, similar to a software product line. Szegletes and Forstner [S36] proposed a reusable framework for adaptive game development. The architecture of the proposed framework consisted of loosely coupled components for better flexibility. They tested their framework by developing educational games. The requirements of the new game must be well aligned with the reusable components of the previously developed game.

Game design document

The Game Design Document (GDD) is an important deliverable in the pre-production phase. It consists of a coherent description of the basic components, their interrelationships, directions, and a shared vocabulary for efficient development. Westera et al. [S37] addressed the issue of design complexity in serious games by proposing a design framework. Furthermore, Salazar et al. [S38] highlighted the importance of a game design document for game development and provided an analysis of many available game design documents from the literature. They also compared their findings with traditional software requirement specifications and concluded that a poor game design document can lead to poor-quality product, rework, and financial losses in the production and post-production phases. Hsu et al. [S40] pointed out the issues of level determination in games and trade-off decisions about them. They proposed an approach to solve the trade-off decision problem, which is based on a neural network technique and uses a genetic algorithm to perform design optimization. Khanal et al. [S41] presented design research for serious games for mobile platforms, and Cheng et al. [S42] provided design research for integrating GIS spatial query information into serious games. Finally, Ibrahim and Jaafar [S43] and Tang and Hanneghan [S44] worked on a game content model for game design documents. Currently, GDD suffers from formalism and incomplete representation; to address this issue, the formal development of GDD is very important. A comprehensive GDD (focused on the game’s basic design and premises) results in good game quality.

Game prototyping

Game prototyping in the pre-production phase helps the developer to clarify the fundamental mechanics of the final game. Game prototyping in the preproduction phases is considered important because it is used to convey game and play mechanics and also helps in evaluating a game player’s experience. Reyno and Cubel [S49] proposed automatic prototyping for game development based on a model-driven approach. An automatic transformation generates the software prototype code in C++. De Silva et al. [S48] proposed community-driven game prototyping. The developer can approach the well-established community and focus on the technical stuff rather than starting from scratch. They used this approach for massive, multi-player online game development. Guo et al. [S50], Kanev and Sugiyam [S51], and Piesoto et al. [S52] proposed analysis of rapid prototyping for Pranndo’s history-dependent games, 3D interactive computer games, and game development frameworks respectively. Prototypes also help to identify missing functionality, after which developers can easily incorporate quick design changes. Model-driven or rapid-prototyping approaches can be used to develop game prototypes.

Design tools

Game design tools are used to help game developers create descriptions of effects and game events in detail without high-level programming skills. Cho and Lee [S56] and Segundo et al. [S57] proposed an event design tool for rapid game development and claimed that it does not require any kind of programming skill. These tools also enable reuse of existing components and reduce the total time of the game-creation process.

Risk management

In the game development domain, risk management factors do not receive much discussion by researchers. Risk management is very important from a project management point of view. Identifying risk factors in the game development process is also important. In game development, the project manager is the game producer and must bring together management, technical, and aesthetic aspects to create a successful game. The study by Schmalz et al. [S58] is the only study highlighting the issue of risk management in video development projects. They identified two risk factors during the development process: failure of development strategy and absence of the fun factor. In game development, important risk factors can be the development strategy, the fun factor or extent of originality, scheduling, budgeting, and others, but very low priority has been given by game developers to formal analysis of risk factors.

4.2.2 Production phase

Asset creation.

Asset creation in the production phase is the foundation stage where game developers create the various assets and then use them in the game implementation phase. In the production phase, the first step is to create assets for the game. One of these assets is audio creation. Migneco et al. [S63] developed an audio-processing library for game development in Flash. It includes common audio-processing routines and sound-interaction Web games. Minovic et al. [S65] proposed an approach based on the model drive method for user interface development, and Pour et al. [S64] presented a brain computer interface technology that can control a game on a mobile device using EEG Mu rhythms. For audio processing, open-source libraries are available, especially for games. Audio and interface design are examples of game assets.

Storyboard production

Storyboard production is the most important phase of game production; it involves development of game scenarios for level solutions and incorporation of artificial intelligence planning techniques for representing the various features of games through a traditional white board or flow chart. Pizzi et al. [S59] proposed a rational approach that elaborated game-level scenario solutions using knowledge representation and also incorporated AI techniques to explore alternative solutions by direct interaction with generated storyboards. Finally, Anderson [S61] presented a classification of scripting systems for serious and entertainment games, and Cai and Chen [S62] explored scene editor software for game scenes. Their approach was based on the OGRE.Net framework and C++ technology. Various scripting editors based on different technologies are available for game developers to produce storyboards. Some of this software helps to develop and edit scenes at different game levels, and other software helps by generating game levels automatically based on a description.

Development platforms

The studies classified under this category proposed various types of platforms for game development. Development platforms provide a ready-made architecture for server–client connectivity and help developers create games quickly. Open-source development platforms are available, but developers must customize them according to the required functionality. Peres et al. [S69] used a scrum methodology for game development, especially for multiple platforms, and implemented interfaces with social networking Web sites such as Twitter and Facebook. Jieyi et al. [S70] proposed a platform for quick development of mobile 3D games. First, the platform implemented the game template in two environments such as the Nokia series 60 platform and the Symbian OS. The second part of the process involved analysis of the entire game structure and extraction of game parameters for later customization. Finally, the tool could be used for game customization. Lin et al. [S] developed intelligent multimedia mobile games from embedded platforms. The proposed communication protocol was able to control the embedded platform to achieve the game usability and amusement. Mao et al. [S78] presented a logical animation platform for game design and development, and Alers and Barakova [S81] developed a multi-agent platform for an educational children’s game. Suomela et al. [S77] highlighted the important aspects of multi-user application platforms used for rapid game development. Some researchers have proposed a development platform similar to that described above that provides connectivity along with client customization and unnecessary updating of game servers.

Formal language description

Game semantics can be classified under formal language description for programming languages; only two studies were reported under this classification. The formal language description of game semantics provided a way to gain insight into the design of programming languages for game development. Mellies [S99] proposed a denotational prepositional linear logic for asynchronous games, and Calderon and McCusker [S100] presented their analysis of game semantics using coherence spaces. Very little work has been reported in this area, and very few game semantic descriptions of languages have been published.

Programming

Code complexity is increasing, especially in game development, because of the incorporation of complex modules, AI techniques, and a variety of behaviors. The most common programming languages used in game development are object-oriented structured languages such as Java, C, and C++. Studies classified under this category explored the programming aspect of game development. El Rhalibi et al. [S82] proposed a development environment based on Java Web Start and JXTA P2P technologies called Homura and NetHomura. It extends the JME game engine by facilitating content libraries, providing a new interface, and also providing a software suite that supports advanced graphical functionalities within IDE. The other two studies, done by Meng et al. [S84] and Chen and Xu [S85], also explored programming languages such as C++, DirectX, and Web GL and also Web Socket technologies for game development. Three studies by Yang et al. [S87], Yang and Zhang [S88], and Wang and Lu [S89] explored collision detection algorithms from a game logic aspect for software games, proposed A* search, and AI optimization-based algorithms.

Wang et al. [S83] proposed a framework for developing games based on J2ME technology. Zhang et al. [S92] also explored the effects of object-oriented technology on performance, executable file size, and optimization techniques for mobile games and suggested that object-oriented technology should be used with great care because the structured programming in game development is highly competitive. Bartish and Thevathayan [S86] and Fahy and Krewer [S90] analyzed the use of agents, finite state machines, and open-source libraries for the overwhelmingly complex process of multi-platform game development. Optimization techniques can be used with object-oriented programming to avoid unnecessarily redundant classes and inheritance, and to handle performance bottlenecks. These languages can be used across different development environments such as Android, iOS, Windows, and Linux. Researchers have proposed various approaches and tools for efficient game development. The integration of various development artefacts into games can also be done by generative programming, which also helps to achieve efficient development.

Game engine

A game engine is a kind of special software framework that is used in the production phase for creating and developing games. Game engines consist mainly of a combination of core functionalities such as sound, a physics engine or collision detection, AI, scripting, animation, networking, memory management, and scene graphs. Hudlicka [S108] identified a set of requirements for a game engine, including identification of the player’s emotions and the social interactions among game characters. This is the only study that has highlighted the important functionalities that an affective game engine must support. Another study by Wu et al. [S109] focused on game script engine development based on J2ME. It divided script engines into two types. The first type is the high-level script engine that includes packaging and refining of the script engine. The second type, the low-level script engine includes feature packages associated only with API. Four studies [S102], [S105], [S106], and [S107] explored the development of game engines on mobile platforms. Finally, Anderson et al. [S109] proposed a game engine selection tool. Recently, developers have been using previously developed or open-source game engines to economize on the game development process. Various researchers have proposed script-based, design pattern-based, and customizable game engines. In the GDSE process life cycle, game engines automate the game creation process and help a developer to develop a game in a shorter time.

Implementation

The foundations of game theory are used in game development because it is a branch of decision theory that describes interdependent decisions. Most studies in this category described different aspects of game implementation technologies on various types of platforms. They considered improving programming skills, 2D/3D animations and graphics, sound engineering, project management, logic design, story-writing interface design, and AI techniques. Various kinds of game implementation technologies can be found in the literature. Vanhatupa [S117] presented a survey of implementation technologies especially for browser games. The technologies explored in these studies are mainly server applications (application runtime, server-side scripting, and user interface and communication), client applications, databases, and architecture. The same study also described the accessories that can be used for implementation: application platforms, game engines, and various types of plug-ins. Abd El-Sattar [S112] proposed an interactive computer-based game framework for the implementation process. The framework includes steps from design through implementation that are based on game theory foundations and focus mainly on game models, Nash equilibrium, and strategies of play. The proposed framework includes architectural design and specifications, a proposed game overview, a game start-up interface and difficulty scaling, game modelling, the game environment and player control, and a free-style combat system.

Four studies [S113], [S114], [S119] and [S120] focused mainly on a development framework for mobile devices. Su et al. [S96] proposed a framework describing implementation of various main modules such as pressure movement, a thread pool based on the I/O completion port, and a message module. They also claimed that their proposed framework addressed the problems of traditional frameworks such as the single-server exhaustion problem, synchronization, and thread-pooling issues. Jhingut et al. [S114] discussed 3D mobile game implementation technologies from both single-player and multi-player perspectives. They also evaluated two game APIs: MDP 2.0 and M3G API. Finally, Kao et al. [S120] proposed a client framework for mobile devices that used a message-based communication protocol and reserved platform-specific data as much as possible. A few researchers have proposed agent-based frameworks as explored above for effective communication and synchronization between system components.

4.2.3 Post-production phase

Quality assurance.

Process validation plays an important role in assessing game quality. Collection and evaluation of process data from the pre-production phase through to the post-production phase either provide evidence that the overall development process produces a good-quality game as a final product or reveal that it cannot. Only two studies were reported under this classification. Stacey et al. [S122] used a story-telling strategy to assess the game development process. They carried out a two-year case study on a four-person development team. Astrachan et al. [S126] tried to validate the game creation process by analyzing the development process and design decisions made during development. The scope of studies done under this category was limited. The case studies were done for small teams and were limited to only one phase. In the game development process, quality assurance and process validation are critical components, and standard methodologies are lacking. More exploration is needed to provide deeper insights. QA for games needs more research attention because very little work has been reported.

Beta testing

Beta testing in games is used to evaluate overall game functionality using external testers. Beta testing is a kind of first public release for testing purposes by users. Game publishers often find it effective because bugs are identified by users that were missed by developers. If any desired functionality is missing, it must be addressed at this stage. This testing is performed before final game release. Under this classification, only four studies [S127], [S128], [S129], and [S130] were reported. Hable and Platzer [S129] evaluated their proposed development framework for mobile game platforms. Omar et al. [S128] evaluated educational computer games and identified two evaluation techniques: Playability Heuristic for Educational Games (PHEG) for expert evaluators, and Playability Assessment of Educational Games (PAEG) for real-world users. The proposed AHP-based Holistic Online Evaluation System for Educational Computer Games (AHP_HeGES) online evaluation tool can be used in the evaluation process. Very little work was reported in this category.

Heuristic-based testing

Heuristics are a kind of design guideline and can be used as an evaluation tool by game design developers or users. Basically, heuristics can be used in software engineering to test the interface. In games, evaluation must extend beyond the interface because other playability experiences also need evaluation such as the game story, play, and mechanics. Six studies [S132], [S133], [S134], [S146], [S147], and [S148] fell under this classification. Al-Azawi et al. [S132] proposed a heuristic testing-based framework for game development. The proposed framework divides testing by two types of user: experts and real-world users. Experts evaluate playability, game usability, and game quality factors. Users evaluate the game as a positive or negative experience. Omar and Jaafar [S133] and Al-Azawi et al. [S134] proposed a framework for the evaluation phase in the game development process. Heuristic testing can be done during the development process and repeated from the early design phase. It is perfect for game testing because after the game is implemented, if anything goes wrong, it will be too expensive to fix and will affect the project schedule. This topic also needs attention by researchers.

Empirical testing

Empirical testing approaches for the game-testing phase have been explored by only a few researchers. The approaches described by these researchers have focused only on final-product quality and usability. Only two studies were reported under this classification [S135] and [S136]. Escudeiro and Escudeiro [S135] used a Quantitative Evaluation Framework (QEF) to evaluate serious mobile games and reported that QEF frameworks are very important in validating educational games and final-product quality. Choi [S136] analyzed the effectiveness of usability-expert evaluation and testing for game development. Experimental results showed the importance of the validation process in game development. The scope of the studies done under this category was very limited, and other aspects of final-product testing have not been explored by researchers.

Testing tools

Development of testing tools has not been addressed by many researchers. Only one study [S137] was reported under this classification. Cho et al. [S137] proposed testing tools for black-box and scenario-based testing. They used their tool on several online games to verify its effectiveness. Tools for game testing facilitate the testing process. The proposed scope of study was also limited, and available testing tools have focused only on evaluation of online games.

After a game has been developed, the final step is marketing. Marketing of games includes a marketing strategy and a marketing plan. The marketing strategy is directly related to the choice of users and the types of games that are in demand. The marketing plan is something that a publisher can give to a distributor to execute on the publisher’s behalf. Some studies have been done from the perspective of game-user satisfaction that provide the baseline for the factors that game developers must take into account for new game development. Yee et al. [S142] described a game motivation scale based on a three-factor model that can be used to assess game trends. Three studies [S139], [S143], and [S144] empirically investigated the perspective of game-user satisfaction and loyalty. No study in the literature has directly captured a marketing strategy and a marketing plan for games.

4.3 RQ 3: What research approaches are being used by researchers in digital game domain?

Table  10 shows that most GDSE process life cycle studies have used an exploratory research approach. Figure  8 shows a comparison between the three research approaches used in the GDSE process life cycle domain. Figure  9 shows a comparison among the empirical research methods used in the GDSE process life cycle domain. The results suggest that surveys are most frequently used in GDSE domain research.

GDSE process life cycle research approaches

Empirical research approaches

These results were to be expected because the GDSE domain has only been growing since 2005; before 2010 more studies follow the descriptive approach because the field was young. After 2010, more studies have followed the exploratory approach because the domain has been maturing. More specifically, exploratory and descriptive approaches seem now to be equally used in the GDSE process life cycle domain.

4.4 RQ4: What empirical research methods are being used in the software games domain?

Table  11 depicts the results of the RQ4. The experimental empirical method is less used in the GDSE process life cycle domain, as mentioned by Wohlin et al. ( 2000 ), because carrying out formal experiments requires significant experience. The case-study method has also been used infrequently by researchers. The reason for this could be that case studies require project data obtained through various types of observations or measurements, and no research database or repository is available for the GDSE process life cycle domain. Finally, the survey method was more common than the other two methods. This is reasonable because the GDSE domain is still immature and researchers are trying to produce knowledge by questioning game users, experts, and others.

5 Conclusions

The GDSE process proved to be incredibly challenging as game technology including game platforms and engines changes rapidly and coding modules are used very rarely in the another game project. However, recent success of digital game industry enforces further stress along with game development challenges and highlights the need of good practices adoption for game development process. In order to find out the specific area in game development software engineering process for improvement, assessment of process activities needs to be performed. However, due to relatively young history and empirical nature of the field, there has not been any development strategies or set of best practices to carry out game development fully explored. This systematic literature review helps to identify the research gaps in game development life cycle.

The main objective of this research was to provide an insight into the GDSE process life cycle domain because, in the past, researchers have pointed out that it is different from the traditional software development process. To achieve this objective, a systematic literature review was performed, which confirmed the first step of the evidence-based paradigm. The results also confirmed that the GDSE process life cycle domain is different from the traditional software engineering development process and that research activity is growing day by day, attracting the interest of more researchers. This observation provided an evidence for developers they need to look for other important activities on top of software development process. This paper describes the various topics in the GDSE domain and highlights the main research activities related to the GDSE process life cycle. The research topics identified in the GDSE were a combination of different disciplines and together they complete the game development process.

The most heavily researched topics were from the production phase, followed by the pre-production phase. On the other hand, in the post-production phase, less research activity was reported. In the pre-production phase, the management topic accounted for the most publications, whereas in the production phase, the development platform, programming, and the implementation phase attracted the most researchers. The production phase has attracted more research because game developers focus more on implementation and programming because of the limited game-development time period. The post-production phase includes process validation, testing, and marketing topics. Very little research activity was observed in this area because the quality aspect of game development is not yet a mature field. These results highlighted that researcher’s need to pay attention especially in the phase of post-production.

In addition to research topics, more researchers used exploratory research methods; as for empirical research methods, surveys were carried out by more researchers than case studies and experiments. Overall, the findings of this study are important for the development of good-quality digital games. Rapid and continual changes in technology and intense competition not only affect the business, but also have a great impact on development activities. To deal with this strong competition and high pressure, game development organizations and game developers must continually assess their activities and adopt an appropriate evaluation methodology. The result of the study highlighted that use of a proper assessment methodology will help the organization identify its strengths and weaknesses and provide guidance for improvement. However, the fragmented nature of the GDSE process requires a comprehensive evaluation strategy, which has not yet been entirely explored. Finally, this kind of research work provides a baseline for other studies in the GDSE process life cycle domain and highlights research topics that need more attention in this area. The findings of this study will help researchers to identify research gaps in GDSE process life cycle and highlights areas for further research contributions. This study also is a part of a larger project aiming to propose a digital game maturity assessment model (Aleem et al. 2016a ). The identified important dimensions are developer’s perspective (Aleem et al. 2016b ), the consumer, the business (Aleem et al. 2016c ), and the process itself. It also reinforces the assertion that the GDSE process life cycle domain is a complex scientific domain comparable to the software engineering development process, and it needs more attention and consideration of different factors in game development software engineering process.

In short, this study presents a systematic literature review of the GDLC topics. Overall, the findings of this study are important for the development of good-quality digital games because they highlight the areas that needs research attention. The results of this study have shown that the fragmented nature of the GDLC process requires a comprehensive evaluation strategy, which has not yet been entirely explored. Finally, this kind of research work provides a baseline for other studies in the GDLC domain and highlights research topics that need more attention in this area. The findings of this study will also help researchers to identify research gaps in the GDLC and highlight areas for further research contributions.

Abbreviations

Game Design Document

Game Development Software Engineering (GDSE)

Quantitative Evaluation Framework

Systematic Literature Review

Ahmed, F., Capretz, L. F., 2011. A business maturity model of software product line engineering. Information Systems Frontiers, Springer, 13, 4, 543–560, DOI: 10.1007/s10796-010-9230-8

Aleem S, Fernando Capretz L, Ahmed F (2016). A Digital Game Maturity Model (DGMM), Entertainment Computing 17, 55-73. http://dx.doi.org/10.1016/j.entcom.2016.08.004

Aleem S, Capretz LF, Ahmed F (2016a) Critical Success Factors to Improve the Game Development Process from a Developer’s Perspective. J Comput Sci Technol 31(5):925–950

Article   Google Scholar  

Aleem S, Capretz LF, Ahmed F, (2016c). Empirical investigation of key business factors for digital game performance, Entertainment Computing, Vol. 13,pp. -25-36, http://dx.doi.org/ 10.1016/j.entcom.2015.09.001

Alvarez, J. Michaud, L., (2008). Serious Games: Advergaming, Edugaming, Training, and More, IDATE

Ampatzoglou A, Stamelos I (2010) Software engineering research for computer games: a systematic review. J Inf Softw Technol Elsevier 52(9):888–901.

Bethke E (2003). Game Development and Production. Wordware game developer's library. Wordware Pub, Plano. ISBN 978-0-585-44833-6

Blitz game studio, (2014). Project Lifecycle. Retrieved May 1, 2014 from http://www.blitzgamesstudios.com/blitz_academy/game_dev .

Boyle EA, Connolly TM, Hainey T, Boyle JM (2012) Engagement in digital entertainment games: A systematic review. Comput Hum Behav 28:771–780

Brereton P, Kitchenham B, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(4):571–583

Budgen D, Turner M, Brereton P, Kitchenham B (2008). Using mapping studies in software engineering. In: Proceedings of Psychology of Programming Interest Group (PPIG). Lancaster University, Lancaster. pp. 195–204

Cai KY, Card D (2008) An analysis of topics in software engineering. J Syst Softw 81(6):1051–1058

Capretz LF, Lee PA (1992) Reusability and life cycle issues within an Object-Oriented Design methodology (refereed). In: Ege R, Singh M, Meyer B (eds) Technology of Object-Oriented Languages and Systems. Prentice Hall, Englewood Cliffs, pp 139–150. ISBN 0-13-042441-2

Google Scholar  

Castillo T, Novak J, (2008). Game Development Essentials: Game Level Design. Delmar Cengage Learning. ISBN: 9781401878641

Chandler HM (2010) Game Production Handbook. Johns and Bartletts, Sudbury

Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM (2012) A systematic literature review of empirical evidence on computer games and serious games. Comput Educ 59:661–686

Dyba T, Dingsoyr T (2008) Empirical studies of agile software development: a systematic review. Information and Software Technology 50(9-10):833–859

Dyba T, Kitchenham BA, Jorgensen M (2005) Evidence-based software engineering for practitioners. Software Magazine. IEEE Computer Society 22(1):58–65

Entertainment Software Association (ESA), (2014). Essential facts about the Computer and Video Game Industry. Entertainment Software Association Available at: http://www.theesa.com/wp-content/uploads/2014/10/ESA_EF_2014.pdf . Accessed on 15 Oct 2015.

Entertainment Software Association (ESA), (2015). Essential facts about the Computer and Video Game Industry. Entertainment Software Association. Available at: http://www.theesa.com/wp-content/uploads/2015/04/ESA-Essential-Facts-2015.pdf . Accessed on 15 Oct 2015.

Flood K (2003) Game Unified Process: GameDev., Available at: http://www.gamedev.net/page/resources/_/technical/generalprogramming/game-unified-process-r1940 . Accessed June 12, 2015

Glass RL, Vessey I, Ramesh V (2002) Research in software engineering: an analysis of the literature. Inf Softw Technol 44(8):491–506

Godoy A, Barbosa E F, (2010). Game-Scrum: An approach to agile game development, Proceedings of SBGames 2010 Computing Track (I. S. F. SC, ed.), Sao Carlos, pp. 292–295, November 8–10, pp. 292–295.

Gredler M. E (1995). Designing and evaluating games and simulations. Behavioral Science. Wiley Online Library, 40, 1 (1995), 76–77

Gredler M. E (2003). Games and simulations and their relationship to learning. Handbook of Research on Educational Communications and Technology, Lawrence Erlbaum, Inc: Mahwah, NJ pp. 571–581.

Hendrick A (2014). Project Management for Game Development. Retrieved 20 May 2014, from http://mmotidbits.com/2009/06/

Kaitilla C (2014). How to learn Ouya Gamdev. Retrieved December 20, 2014, from http://gamedevelopment.tutsplus.com/articles/how-to-learn-ouya-gamedev--gamedev-9197 .

Kanode C M., Haddad H M (2009). Software engineering challenges in game development. In Proceedings of the 2009 Sixth International Conference on Information Technology: New Generations, (April 27–29, 2009), 260–265

Keith C (2010) Agile game development with Scrum. Addison-Wesley, Boston

Kitchenham B (2004). Procedures for performing systematic literature reviews. Joint Technical Report. Computer Science Department, Keele University, July 2004, 33 pages.

Kitchenham B, Charters S, (2007). Guidelines for performing systematic literature reviews in software engineering, Software Engineering Group, Keele University and Department of Computer Science, University of Durham, United Kingdom, Technical Report EBSE-2007-01, 2007

Kitchenham, B., Sjoberg, D.I.K., Brereton, P., Budgen, D., Dyba, T., Host, M., Pfahl, D., Runeson, P., 2010. Can we evaluate the quality of software engineering experiments? In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, 1–8

Kruchten P (2000) The Rational Unified Process: An Introduction, 2nd edn. Addison Wesley Longman, Reading

Liming D, Vilorio D (2011). Work for play: Careers in video game development, Occupational Outlook Quarterly. Available at: http://www.bls.gov/careeroutlook/2011/fall/art01.pdf . Accessed on: 30 Sept 2015.

McGrath J (2014). The game development lifecycle: A theory for the extension of the agile project methodology. http://blog.dopplerinteractive.com/2011_04_01_archive.html . Accessed 1 May 2014

McShaffry M (2003) Game coding complete. Paraglyph Press, AZ, USA

Munassar N, Govardhan A (2010) A Comparison Between Five Models Of Software Engineering. International Journal of Computer Science Issues 7(5):94–101

Nayak M (2013). A look at the $66 billion video-games industry, Reuters, Retrieved June 2013 from http://in.reuters.com/article/2013/06/10/gameshow-e-idINDEE9590DW20130610 . Accessed 12 Sept 2014

Newzoo Game Market Research, 2015. Global Report: U.S. and China take half of $113 bn game market in 2018. Available at: http://www.newzoo.com/insights/us-and-china-take-half-of-113bn-games-market-in-2018/ . Accessed 2 Oct 2015

Osbourne-O'Hagan A, Coleman G, O'Connor RV (2014) Software development processes for games: a systematic literature review. In: 21st European Conference on Systems, Software and Services Process Improvement EuroSPI, Luxembourg, 25-27 June 2014

Petrillo F, Pimenta M, Trindade F, Dietrich C (2009) What went wrong? A survey of problems in game development. Computers in Entertainment. ACM Digit Library 7(1(13)):1–22

Plass-Oude Boss, D., Reuderink, B., Van De Laar, B.L.A., Gurkok, H, Muhl, C., Poel, M., Heylen, D.K.J., Nijholt, A. (2010), Human-Computer Interaction for BCI Games: Usability and User Experience. In Proceedings of the International Conference on CYBERWORLDS, A. Sourin (eds), IEEE Computer Society, Los Alamitos, 277–281

Pressman RS (2001) Software engineering: a practitioner approach, 5th edn. Wiley, New York

PWC global media and entertainment outlook 2011–2014, 2011. Available at http://www.pwc.com/gx/en/global-entertainment-mediaoutlook/territory-segments-digital-forecast-overview.jhtml . Accessed on 28 Jul 2013.

Ramadan R., Widyani Y, (2013). Game development life cycle guidelines. In Proceedings of 5th International Conference on Advanced Computer Science and Information Systems (ICACIS). IEEE Computer Society, Jakarta, Indonesia, (September 28–29, 2013) 95–100.

Rieber LP (2005) Multimedia learning in games, simulations and microworlds. Cambridge Handbook of Multimedia Learning. Cambridge University Press, UK, pp 549–567

Book   Google Scholar  

Robin S, (2009). Introduction to game development, 2nd edition. Charles River Media. ISBN-10: 1584506792

Salen K, Zimmerman E (2003). Rules of Play: Game Design Fundamentals. MIT Press, ACM Digital Library. p. 80. ISBN 0-262-24045-9

Schwaber K, Beedle M (2002) Agile Software Development With Scrum. Prentice-Hall, Upper Saddle River

MATH   Google Scholar  

Shadish WR, Cook TD, Campbell DT (2002). Experimental and Quasi-experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, Boston

SUPERDATA 2015 Digital Good Measurement Blog. Worldwide digital games market. Available at: https://www.superdataresearch.com/blog/us-digital-games-market/ . Accessed 30 Dec 2015.

Wohlin C, Runeson P, Host M, Ohlsson MC, Regnell B, Wesslen A (2000) Experimentation in Software Engineering. Kluwer Academic Publishers, Boston/Dordrecht/London

Book   MATH   Google Scholar  

Download references

Authors’ contributions

SA designed the study and performed the review methodology, collected the data, analyzed the data and drafted the manuscript. LC helped to conceive the study and provided guidance to carry out the quality assessments of paper, reviewed the drafted manuscript and fine-tune the final draft. FA helped in study design, provided guidance to present the analysis and helped to draft the manuscript. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Author information

Authors and affiliations.

College of Technological Innovation, Zayed University, Abu Dhabi, 144534, United Arab Emirates

Saiqa Aleem

Department of Electrical & Computer Engineering, University of Western Ontario, London, ON, N6A 5B9, Canada

Luiz Fernando Capretz

Department of Computing Science, Thompson Rivers University, Kamloops, BC, V2C 0C8, Canada

Faheem Ahmed

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Saiqa Aleem .

Additional files

Additional file 1:.

Appendix A (DOC 114 kb)

Additional file 2:

Appendix B (DOC 313 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Aleem, S., Capretz, L.F. & Ahmed, F. Game development software engineering process life cycle: a systematic review. J Softw Eng Res Dev 4 , 6 (2016). https://doi.org/10.1186/s40411-016-0032-7

Download citation

Received : 16 March 2016

Accepted : 25 October 2016

Published : 09 November 2016

DOI : https://doi.org/10.1186/s40411-016-0032-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Software Game
• Online game
• Systematic review
• Development software engineerong proces

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Requirements Engineering Techniques in Software Development Life Cycle Methods: A Systematic Literature Review

2018, International Journal of Advanced Research in Computer Engineering & Technology (IJARCET)

Requirements engineering (RE) is the comprehensiveness of user requirement. It comprises of all processes of life-cycle in identification, specification, analysis, development, validation and evaluation of the requirement. This ensures that the system satisfy the requirement of the customer. Software Development Lifecycle (SDLC) aim to produce a high-quality systems that meets customer expectations, delivered within specific time, able to minimize cost and maximize profit. The SDLC methods considered literature on using Spiral Method, Waterfall Method, Verification and Validation (V-Model) and Rapid Application Development (RAD). However, researcher has not been able to select and review literature that focus on this research work. This work uses method that is proposed by Kitchenham to conduct a Systematic Literature Review (SLR). Some of the reviewed papers considered are from Research Gate, Science Direct, Springer, Institute of Electrical and Electronics Engineers (IEEE), Association of Computing Machinery (ACM) and World Wide Web (WWW). 385 published papers where identified, 27 of the papers are relevant to RE used in SDLC which provide information about the topic. The paper systematically review literatures that will help researchers and practitioners to better understand RE techniques that are used in SDLC.

Related Papers

2014 8th. Malaysian Software Engineering Conference (MySEC)

Masitah Ghazali

International Journal of Soft Computing and Engineering

Kanos Matyokurehwa

Requirements engineering is a torrid task to requirements engineers because requirements keep changing and this affect the project's delivery schedule and cost. Although various authors proposed numerous techniques to be used in requirements engineering, software projects still fail. The issue now lies on which technique to use to minimize project failures. The aim of the study was to identify gaps in requirements engineering techniques used. The paper used a systematic literature review of requirements engineering techniques used from January 2000 to July 2016. The study found out that a lot of techniques are used in requirements engineering and some of the techniques used are not adequately addressing the problem space but the solution space. The study identified some gaps in requirements engineering techniques that need further research in order to solve those gaps.

Dr-Muhammad Suhaib

In this research, we investigate and Analysis of the Requirement Engineering in Software Development Life Cycle and its Systematic Requirements Elicitation in the software development process. In the next step, the research considered some approaches which are previously being used for requirement elicitation. Requirement elicitation has always played a vital role in project development. This research narrows down some pros and cons of RE. Furthermore, it has some research methods, research strategies, and research design which are followed during our research process. The main goal of this research is to introduce a new methodology for requirement gathering and apply it to a case study. Some future work and conclusions of using previous methods and a new approach are also part of this paper.

2011 Ninth International Conference on Software Engineering Research, Management and Applications

Abstract Requirements engineering activities act as a backbone of software development. The more efforts devoted during requirements engineering activities guarantee a better software product. Appropriate selection of requirements has been a challenge for software ...

ZILLE SUBHAN

Software Engineering is a significant domain of Computer Science that deals with all the activities associated with the software development. Basically, a software is developed in a number of phases and each phase is closely linked to all other phases. The success and failure of a phase can heavily affect the other phases. Both requirements analysis and software design are significant phases of the software development process. In fact, the successful completion of a software development task heavily depends on the successful completion of these two phases. This paper is a comparative study of requirements engineering and design phases of different software development approaches. The major objective of this research is to present a detailed analysis of requirements and design phases of traditional and agile software development approaches.

IRJET Journal

Present complexities and higher level of client's expectations from an application leads to software development process to be more alignment towards technical specialists and managerial techniques. Software progression is incessantly put into practice by practitioners in order to meet up the dynamic stakeholder's requirements. Due to globalization, software and technology both become a fragment of any automated entity. Every organization body supposes to build a decent and reliable working software. It has also been inspected that for successful software system, requirements engineering is very crucial phase. This makes a growth in the scope of the requirements engineering process, apart with new challenges of gathering, prioritizing and preprocessing the requirements. Requirements Engineering is well-thought-out as a collection of processes that functions on different levels, which includes organizational, product and project level. In any field, building an application is a thought-provoking task due to the absence of right requirements engineering technique and due to developer's understandability of the actual needs of the client. There are various tools are available to gather requirements from the clients. The major faults are occurred in choosing the right requirements engineering technique which is considered a delicate task, any faults or wrong selection may lead to major catastrophic for the final product. In this paper, we have proposed a requirement engineering process model that produce quality requirements for software development. The proposed framework tries to bridge the gap between theoretical and practical aspects of requirements engineering process. It helps the software experts and analyst to choose a right technique for selected requirements engineering phase. We have thoroughly comprehended and evaluate all the existing requirements engineering techniques with respect to analyst preferences, client experiences, project attributes, software process model characteristics. The successful implementation of proposed requirements engineering process can have a good impression on the creation of qualitative software product.

Armin Eberlein

Talat Ambreen

Requirements Engineering (RE) is recognized as one of the critical phases in software development. RE has its own journals and conferences where lots of work has been published. As the area is maturing, increasingly large numbers of empirically supported studies have been reported in RE. There is a need to synthesize evidence based RE literature. We plan to systematically investigate evidence based RE studies to see and report state of the art in evidence based RE reported research. This paper aims at providing a systematic literature review (SLR) protocol to describe a process for synthesizing the empirically supported work in the area of RE that will eventually present a state of the art of the field. This SLR intends to not only summarize the empirical data regarding RE but will also be helpful for various practitioners in this field to find out areas of RE rich in terms of tools, techniques, frameworks, models and guidelines to aid in their work. It will also facilitate RE resea...

Journal of Software Engineering & Intelligent Systems

The domain of requirements engineering (RE) is intercepted by the interest of a wider community in the software industry. The role of RE is inevitable in the success and failure of software projects. RE is a multi-dimensional area and it is comprised of stakeholder analysis, requirements elicitation, and requirements prioritization mainly. In RE, the valuable requirements are explored from a critical set of stakeholders and these requirements are prioritized in order to design a system of high assurance. Several systematic literature reviews (SLR) are written related to RE domain which deliver a comprehensive knowledge in the domain of RE. Hence, currently no SLR exists that is able to provide an aggregate knowledge about RE intricacies. In this research, the different systematic reviews were investigated in order to aggregate the generated knowledge in different sub-domains of RE. This research highlights the objectives, sub-objectives of the existing SLRs and also presents information about RE issues in different perspectives. Initially, 214 studies were identified and only 39 studies were included in this SLR for research and analysis purposes.

RELATED PAPERS

Ediany Souzah

Klinische Wochenschrift

Wolfgang Kiessling

Milcho Todorov

Rebecca Reed-jones

Journal of International Oral Health

Darmawan Setijanto

Ni Nyoman Suryani

Studia Islamika

Jajat Burhanudin

The B.E. Journal of Economic Analysis & Policy

Howard Wall

Journal of Basic & Applied Sciences

Sadaf Memon

Revista Médica del …

Alvaro Ronco

Global and Planetary Change

Yongkang Xue

Annals of Agricultural & Crop Sciences

Ukesh Darai

Computing Research Repository

Kevin McDaid

The Journal of Physical Chemistry B

Antonio Pizzirusso

Journal of Cereal Science

Sladjana Zilic

Felix San Vicente

Journal of Industrial and Engineering Chemistry

Mohammed Omer

Scientific reports

Nathalie Escaravage

Biochemical and Biophysical Research Communications

Tur-fu Huang

Ultrasonics Sonochemistry

Grigorij Kogan

Andrea Coraddu

Southern Medical Journal

Dorothy Shulman

Bintal Amin 19630403 198803 1 003

2010 Annual IEEE India Conference (INDICON)

Chayanika Bose

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

Evaluation in Life Cycle of Information Technology (ELICIT) Framework: Supporting the Innovation Life Cycle from Business Case Assessment to Summative Evaluation

Polina v. kukhareva.

Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA

Charlene Weir

Guilherme del fiol, gregory a. aarons.

Department of Psychiatry, UC San Diego ACTRI Dissemination and Implementation Science Center, UC San Diego, La Jolla, CA, USA

Teresa Y. Taft

Chelsey r. schlechter.

Department of Population Health Sciences, Center for Health Outcomes and Population Equity, Huntsman Cancer Institute, University of Utah, Salt Lake City, UT, USA

Thomas J. Reese

Department of Biomedical Informatics, Vanderbilt University, Nashville, TN, USA

Rebecca L. Curran

Claude nanjo, damian borbolla, catherine j staes.

College of Nursing, University of Utah, Salt Lake City, UT, USA

Keaton L. Morgan

Heidi s. kramer, carole h. stipelman.

Department of Pediatrics, University of Utah, Salt Lake City, UT, USA

Julie H. Shakib

Michael c. flynn.

Department of Family & Preventive Medicine, University of Utah, Salt Lake City, UT, USA

Kensaku Kawamoto

AUTHOR CONTRIBUTIONS

Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation.

The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies.

The resulting E valuation in Li fe C ycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: 1) business case assessment; 2) stakeholder requirements gathering; 3) technical requirements gathering; 4) technical acceptability assessment; 5) user acceptability assessment; 6) social acceptability assessment; 7) social implementation assessment; 8) initial user satisfaction assessment; 9) technical implementation assessment; 10) technical portability assessment; 11) long-term user satisfaction assessment; and 12) social outcomes assessment.

Discussion:

Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation.

Conclusion:

As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

INTRODUCTION

As the adoption of electronic health records (EHRs) has expanded rapidly in recent years, EHRs have taken center stage in modern healthcare. Moreover, EHRs have become platforms for the delivery of various health interventions. For the purposes of this paper, we define EHR-integrated innovations as health interventions that 1) have a software component , including clinical decision support (CDS) tools (e.g., EHR-integrated alerts and reminders, order facilitators, integrated information displays), specific modules of an EHR system (e.g., for order entry or clinical documentation), and interoperable externals apps; 2) are integrated with the EHR , which could be achieved through various mechanisms such as using EHR native functionality, proprietary interfaces, and/or standards-based interfaces; 3) address a specific healthcare domain and/or process , such as colorectal cancer screening, medication ordering, or neonatal bilirubin management; and 4) aim to improve healthcare outcomes such as quality of patient care, provider experience, or healthcare value. Examples of EHR-integrated innovations include provider and patient reminders for preventive care procedures, an evidence-based institutional order set for community acquired pneumonia, and an interoperable external app supporting shared-decision making for lung cancer screening.

EHR-integrated innovations have been identified as having great potential for improving health and health care. 1 However, to achieve their full potential, EHR-integrated innovations must meet multiple requirements. Ideally, EHR-integrated innovations must integrate into an existing EHR system; meet clinical needs as defined by organizations, national bodies, and/or professional societies; be acceptable to end users including clinicians and patients; integrate seamlessly into existing clinical workflows and information systems; alter clinician or patient behavior; do no harm; and be portable across EHR platforms. The need to meet these various requirements makes development, implementation, and evaluation of EHR-integrated innovations quite challenging. Recent publications show that EHR-integrated innovations do not always meet these requirements. 2 - 4 Thus, it is important to ensure the evaluation of EHR-integrated innovations is supported by an evaluation framework that can help address these various requirements.

Given the complexity of EHR-integrated innovations, we have identified three main requirements for an evaluation framework for EHR-integrated innovations: 1) support the full information technology (IT) life cycle; 2) address measures at society, user, and IT levels; and 3) address issues specific to EHR integration. Justification for these requirements is shown in Table 1 .

Evaluation Framework Requirements

EHR – electronic health record, IT – information technology.

While many comprehensive evaluation frameworks for health information systems have been described in literature reviews, 12 - 17 none of them meet the three requirements summarized in Table 1 . Most evaluation frameworks developed for health information interventions do not meet the first requirement (i.e., support the full IT life cycle). 11 , 18 - 21 For example, the Health Information Technology Research-based Evaluation Framework (HITREF) developed by Sockolow et al. describes 6 evaluation concepts (i.e., structural quality, quality of information logistics, unintended consequences/benefits, effects on quality process, and barriers and facilitators to adoption). 18 , 19 While HITREF acknowledges the importance of context, it does not address the “how” and “when” of evaluation throughout the IT life cycle phases. 15

We identified a few evaluation frameworks aligned with the IT life cycle, 22 - 29 but these frameworks did not meet the second requirement (i.e., address society, user, and IT-level measures). Among these evaluation frameworks, many focused mainly on one level such as society-level, 25 user-level, 22 , 23 or IT-level measures. 24 Finally, the evaluation frameworks that did focus on multiple IT life cycle phases and multiple levels either omitted one of the IT life cycle phases or one of the measure levels. 26 - 29

Moreover, all existing evaluation frameworks failed the third requirement (i.e., address EHR integration issues). More specifically, the existing frameworks do not address the quality of back-end EHR integration, user-interface EHR integration, and the changes in EHR workflows caused by the innovation. 17 Hence, there is a need for a framework describing an appropriate evaluation for every phase of the EHR-integrated innovation life cycle on society, user, and IT levels.

To address this emergent need for a holistic approach to the evaluation of EHR-integrated innovations, we developed the E valuation in Li fe C ycle of IT (ELICIT) framework by incorporating concepts from human factors engineering, software engineering, and social sciences domains across the entire IT life cycle. The purpose of the ELICIT framework is to guide evaluation efforts during the planning, development, implementation, and operation of EHR-integrated innovations. The intended users of the framework are innovation research and development teams such as those from academic health systems and industry.

The objective of this paper is to describe an evaluation framework for supporting the whole life cycle of EHR-integrated innovations.

Definitions

To enable shared understanding across stakeholders grounded in different domains with divergent terminologies, we define the following terms for the purpose of this paper.

Study Design

This work is presented as a case study of the development of a new evaluation framework derived from the review of theoretical frameworks, the convening of an expert panel, and multiple real-world evaluations. All included projects were approved or exempted as quality improvement by the University of Utah (UU) Institutional Review Board (IRB).

This work was conducted at UU Heath, an academic medical center with five hospitals and 12 community clinics. UU Health has used the Epic® EHR in its primary care clinics since 1999 and enterprise-wide since 2016.

As interoperability standards and application programming interfaces (APIs) were increasingly required by government regulations and adopted by healthcare systems and EHR vendors, 35 UU Health launched the ReImagine EHR initiative in 2016 to harness the promise of interoperable apps based on standards such as Substitutable Medical Applications, Reusable Technologies on Fast Healthcare Interoperability Resources (SMART on FHIR). 36 - 39 The ReImagine EHR initiative developed, implemented and evaluated over ten EHR-integrated innovations (e.g., to support neonatal bilirubin management guidelines, lung cancer screening guidelines, smoking cessation, and chronic disease management guidelines), established productive corporate partnerships, and secured over $35 million in grant funding. 40 ReImagine EHR’s innovations are seamlessly integrated with the Epic® EHR; are supported by native CDS components such as preventive care reminders; and are designed using interoperability standards to enable integration with other EHR products and dissemination across healthcare organizations. The initiative is led by the Associate Chief Medical Information Officer (KK). The technical development team consists of nine core team members and dozens of project-based contributors. 40

Evaluation Infrastructure

When initially established in 2016, the ReImagine EHR initiative focused primarily on the design and development of EHR-integrated innovations. The Director of the UU Department of Biomedical Informatics Sociotechnical Core (CW) oversaw the evaluation aspects of this new initiative, with support provided by seed research grants. Given the number of requested innovations and resource constraints, only limited evaluation was performed. However, as more innovations entered clinical use, we recognized the need to invest further in evaluation and to establish a formal evaluation team. In 2018, the initiative hired an evaluation scientist (PVK) with training in biomedical informatics, biostatistics, health technology assessment, and data science to oversee and coordinate the data-focused aspects of evaluation with statistical support from the UU Study Design and Biostatistics Center. Other key members of the evaluation team include project leads, clinical informaticists, human-factors engineers, software engineers, clinical champions, and implementation scientists. The engagement of these stakeholders is supported by a combination of operational funding, training grants, research grants, and industry collaborations.

Framework Development

Overall approach to framework development.

The four-year iterative process of framework development is summarized in Figure 1 . Our evaluation framework was iteratively developed through focused literature review, practical experience evaluating innovations, and expert consensus. Practical experience included the evaluation of traditional EHR-integrated innovations 41 - 46 as well as interoperable EHR-integrated innovations 40 , 47 - 51 including a neonatal bilirubin management app, 47 , 48 a diabetes pharmacotherapy outcome prediction app, 49 a chronic disease management app, 50 a lung cancer screening shared decision app, 51 and a clinical calculator app. 52 These projects involved all the clinical informaticists noted below who worked together developing and evaluating the applications. The expert consensus was based on this shared experience and was formally reached through five meetings in 2020 where the model went through discussion, review, and further development each time. Additional asynchronous review and feedback of the framework continued in 2021 involving 10 experts with training in biomedical informatics (PVK, CW, GDF, TT, TJR, HSK, KK), human factors engineering (TT, HSK, CW), software engineering (GDF, HSK, KK, CN), implementation science (GAA, CRR), and biostatistics/effectiveness research (PVK, CW). During the meetings PVK presented the framework draft, asked specific questions about the model and elicited experts’ feedback regarding relevant outcomes, questions, methods and theoretical perspectives. Based on the outcomes of each meeting, PVK revised the framework for further review by these experts. After 5 meetings and the achievement of initial consensus, we transitioned to individual meetings and asynchronous communication until we reached consensus on the final framework.

Framework Development Method and Timeline

Approach to Defining the IT Life Cycle Phases

We defined the IT life cycle phases based on two sources: Ammenwerth et al. ’s planning, development, implementation, and operation phases 6 and the Exploration, Preparation, Implementation, Sustainment (EPIS) implementation science framework phases. 7 , 32 , 53 The EPIS framework describes four life cycle phases for the implementation of health services interventions. 53 In the Exploration phase, stakeholders consider emergent or existing health needs and identify the best guidelines or clinical interventions to address those needs. 53 In the Preparation phase, the potential barriers and facilitators of implementation are identified for external (outer) and organizational (inner) contexts, adaptation needs are further assessed for both the innovation (e.g., innovation) and for system or organizational structures or processes (e.g., reimbursement, changes in clinic workflows), and a detailed implementation plan is developed. 53 In the Implementation phase, the intervention is rolled out into clinical practice. In the Sustainment phase, the intervention continues to be delivered, with or without adaptation as needed. 53

Approach to Defining the Society, User, and IT-Level Measures

In creating our evaluation framework, we sought to incorporate key insights, theoretical perspectives, and practices from health informatics as well as inter-related domains of social sciences, human factors engineering, and software engineering ( Figure 2 ). Relevant frameworks, models, theories and reporting guidelines are described below in the context of three levels of EHR-integrated innovations: society, user, and IT.

Overlapping Domains Informing Evaluation Studies in Health Informatics

Society-Level Measures

To select exemplar society-level measures, we focused on two specific social sciences: effectiveness research and implementation science. Effectiveness research is the systematic evaluation of health technology to demonstrate the impact and inform stakeholders’ decision making related to the dissemination of health technologies based on their effectiveness. 54 Effectiveness research perspectives are a loose collection of evaluation models that include clinical trials, heath technology assessment, comparative effectiveness research, health services research, health economics and outcome research methodologies. 55 Effectiveness research is critical for identifying return on investment and summative assessments of process, health, 21 , 54 economic, 56 and safety outcomes. 57 - 59 Reporting guidelines in effectiveness research include Consolidated Standards of Reporting Trials (CONSORT) 60 , Standards for QUality Improvement Reporting Excellence 2.0 (SQUIRE 2.0), 61 STAtement on Reporting of Evaluation studies in Health Informatics (STARE-HI), 62 and Consolidated Health Economic Evaluation Reporting Standards (CHEERS). 63

A specific branch of social sciences called Implementation Science is especially useful for evaluating society level measures. Implementation science is “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services”. 64 Implementation science evaluation frameworks assess implementation strategies and outcomes such as adoption, reach and implementation fidelity. 7 , 31 , 32 , 53 , 65 , 66 The main evaluation frameworks in this domain include Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM), 65 Nonadoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS), 67 and the Implementation Outcomes Framework. 66 Process-oriented frameworks such as EPIS are useful in describing processes such as implementation or development. 7 , 32 , 53 Determinant frameworks such as the Consolidated Framework for Implementation Research (CFIR) 31 describe multi-level factors which influence implementation outcomes. 66 Reporting guidelines in implementation science domain include Standards for Reporting Implementation Studies (StaRI) 68 and Framework for Reporting Adaptations and Modifications to Evidence-based interventions (FRAME). 30 Traditionally, implementation science frameworks have focused on the implementation of social and health services interventions, 65 , 69 with less emphasis on digital technologies. However, there is an increasing demand for the adoption of implementation science frameworks in innovation development and evaluations. 70 - 72

User-Level Measures

To select exemplar user-level measures, we focused on human factors engineering research. Human factors engineering entails studying user’s characteristics, needs, abilities, and limitations, and designing software around the user needs. Commonly-used human factors theories aiming to understand and modify end user behavior include the Social Cognitive Theory (SCT), 73 the Situational Awareness (SA) framework, 74 and the Unified Theory of Acceptance and Use of Technology (UTAUT). 75 User centered design includes three steps. 22 , 23 , 76 - 79 First, users’ needs, tasks, values, and workflows are identified using observations, interviews, or focus groups. These activities are grounded in models of contextual design, 80 targeted Cognitive Task Analyses, 81 , 82 and representational analysis to identify the informational display formats that are meaningful to users. 83 Second, low and high fidelity prototypes are designed through an iterative process that involves journey mapping, story boarding, or concept mapping. Third, usability testing is conducted on the prototypes. 22 , 84 , 85

IT-Level Measures

To select exemplar IT-level measures, we focused on the software engineering domain. Software engineering entails conceiving, specifying, designing, programming, documenting, testing, and refining software. Software engineering evaluation activities in health informatics address the soundness of the software architecture, 24 functionality, 86 security, 87 interoperability, 48 , 88 and readiness for clinical use. 89 Process-oriented frameworks in this domain include models for the software development life cycle. 90 The DeLone and McLean Information Systems Success Model specifies system quality, information quality and service quality as the main indicators of success at the technology level. 91

Example: Representative EHR-integrated Innovation Project

We illustrate use of the ELICIT framework by describing the evaluation of a representative innovation – the Neonatal Bilirubin Management App. 47 , 48 This innovation focuses on implementation of the 2004 American Academy of Pediatrics guideline for identifying and treating newborns with elevated bilirubin levels, which is a common condition affecting newborns that can lead to permanent brain damage. 92

ELICIT Framework

The resulting ELICIT framework can support the whole IT life cycle. The first element of the ELICIT framework is the four often-overlapping and iterative IT life cycle phases: I. Planning, II. Development, III. Implementation, and IV. Operation ( Figure 3 ). 7 , 32 , 93 In the Planning phase, the software idea is conceived and prioritized, and the software development decision is made. In the Development phase, the software is designed and developed, the software rollout decision is made, and governance approval is secured. In the Implementation phase, the implementation is planned and the software is gradually rolled out into clinical practice. In the Operation phase, the software is maintained, updated as needed and disseminated to other clinical sites.

The IT Life Cycle Phases

The ELICIT framework describes three levels of measures: society, user, and IT, which could be remembered with an acronym SUIT. Examples of measures corresponding to different levels are displayed in Figure 4 . Society-level measures include impact on patient health, organizations, business processes, and return on investment. User-level measures focus on user experience, IT usability, and EHR workflows. Finally, IT-level measures focus on system, information, and service quality.

Examples of Measures at the Society, User, and IT Levels

The ELICIT framework is shown in Figure 5 and expanded in Tables 2 -5. The ELICIT framework consists of 12 evaluation steps fully covering the four phases of the IT life cycle and society, user, and IT measure levels.

ELICIT Framework Overview: IT Life Cycle Phases, Levels of Measures, and Evaluation Steps

Planning Phase: Evaluation Steps, Exemplar Questions, and Exemplar Methods

While evaluation is an iterative process and one in which different activities may occur in parallel, we provide in Figure 5 a typical progression across the 12 evaluation steps of the ELICIT framework. In the planning phase, evaluation starts with the problem being addressed and moves from social needs to user needs to more specific technical requirements. In the development phase, the direction of evaluation is generally reversed, beginning with verifying that the prototype meets minimum technical requirements then progressing to user acceptance testing and evaluation of intention to use by the intended social group. The implementation phase then generally starts with the assessment of organizational readiness for the innovations, moves on to evaluating initial user satisfaction, and the technical implementation is monitored as it is deployed at a larger scale and enhancement requests are implemented. Finally, the operation phase includes the technical assessment of the innovation to determine whether it could be disseminated to other health systems, as well as the evaluation of long-term user satisfaction and the impact of the innovation on clinical and financial outcomes important to society.

ELICIT framework steps, potential evaluation questions, and methods for data acquisition and analysis are included in Tables 2 -5. While depicted in a linear form, steps within phases can occur in parallel or in different orders, and projects may move back to earlier phases (e.g., to take an implemented innovation back to software design and development based on user feedback).

The proposed questions and methods are labeled as “Exemplar Questions” and "Exemplar Methods" as they are clearly not comprehensive. The listed methods are a subset of approaches from a much larger superset of potential questions and methods, as there are many factors beyond those discussed in the paper that should be considered when selecting specific evaluation questions and methods. Actual questions and methods should be selected based on societal and user needs discovered during the planning phase and adapted based on available resources.

The planning phase involves innovation idea conception, prioritization, coordination with stakeholders, requirement gathering, and project governance review and approval. ELICIT framework steps 1, 2 and 3 fall into this phase ( Table 2 ).

The development phase involves software design, software development, and governance review and organizational approval of the implementation. ELICIT framework steps 4, 5, and 6 fall into this phase ( Table 3 ).

Development Phase: Evaluation Steps, Exemplar Questions, and Exemplar Methods

EHR – electronic health record, HIPAA – Health Insurance Portability and Accountability Act, NASA TLX - National Aeronautics and Space Administration Task Load Index, PHI – protected health information.

The implementation phase involves implementation strategy design, staff communication, education and training, and pilot innovation rollout followed by wider rollout with iterative innovation improvements as needed. ELICIT framework steps 7, 8, and 9 fall into this phase ( Table 4 ).

Implementation Phase: Evaluation Steps, Exemplar Questions, and Exemplar Methods

CFIR – Consolidated Framework for Implementation Research, EHR – electronic health record, EPIS - Exploration, Preparation, Implementation and Sustainment, FRAME – Framework for Reporting Adaptations and Modifications to Evidence-based interventions, SCT – Social Cognitive Theory, SUS – system usability scale, UTAUT - Unified Theory of Acceptance and Use of Technology.

The operation phase involves conducting a clinical trial to assess effectiveness, maintaining and monitoring innovations after the changes in the innovation have stabilized and disseminating the innovations to other sites. ELICIT framework steps 10, 11, 12 fall into this phase (Table 5).

Example: Using ELICIT Framework for a Representative EHR-integrated Innovation

We describe below use of our evaluation framework for a representative innovation, the Neonatal Bilirubin Management App. 47 , 48

I. Planning phase

Step 1. business case assessment.

The business case for this innovation was to improve the management of neonatal hyperbilirubinemia. 92 The healthcare guideline to address neonatal hyperbilirubinemia has been available since 2004 and was adopted as the standard of care at the UU Health newborn nursery shortly thereafter. 92 However, there was wide variability in guideline adherence among the providers. To reduce this variability, universal bilirubin screening was implemented at UU Health in 2016, and an external standalone Web tool known as BiliTool was used to assess whether bilirubin levels required treatment by manually entering patient data abstracted from the EHR. The goal of the innovation project was to improve the fidelity of the guideline implementation and to save provider time through automation. Other motivators for pursuing this project included a prototype solution available through the SMART gallery app store, 105 a manageable scope, the engagement of enthusiastic clinical champions (JHS and CHS), and the availability of key data requirements in the EHR. Moreover, no clear alternatives were available for fully meeting stakeholder needs within the EHR.

Step 2. User requirements gathering

The user requirements were gathered through Cognitive Task Analysis with clinical champions performed by JHS, CHS, and KK. The provider tasks and decisions included gathering relevant EHR data about mother-infant dyad; assessing newborn bilirubin levels in relation to patient-specific thresholds; assessing the probability of rebound hyperbilirubinemia following phototherapy; 106 and making guideline-based care decisions. 92

Step 3: Technical requirements gathering

The technical requirements were identified through the analysis of business and user requirements by KK and other members of the ReImagine EHR software development team. Key identified requirements included app loading times of under 5-10 seconds and limiting the data elements to the US Core FHIR APIs as much as possible to improve interoperability.

II. Development phase

Step 4. technical acceptability assessment.

Technical acceptability was ensured through automated regression testing and manual integration testing in the test EHR environment to assess accuracy and response time performed by KK and the software development team. As is the case with all of our innovations proceeding to clinical use, the security risks were formally reviewed by the Information Security Office, and these risks were deemed by the institutional leadership to be outweighed by the project benefits.

Step 4. User acceptability assessment

User acceptability assessment was conducted through multiple rounds of user feedback on low- and high-fidelity prototypes developed by the software development team. Heuristic evaluation based on Neilsen’s design principles 84 was also used to identify and address usability issues by TT, PVK, HSK, and CW.

Step 6. Social acceptability assessment

For the social acceptability assessment , we focused on determining the efficiency gains. The efficiency assessment was conducted through a randomized, controlled, IRB-approved experiment in which 12 pediatric resident physicians were asked to manage bilirubin in 2-4 newborns, who were randomized to receive care supported by our EHR-integrated innovation or the previous standalone alternative (BiliTool) performed by PVK, HSK, and CW in 2018. This study showed a time savings of 66 seconds per bilirubin assessment (95% CI, 53-79) compared with usual time required for neonatal bilirubin management. 47

III. Implementation phase

Step 7. social implementation assessment.

Implementation factors assessment was done with clinical champions and the EHR team by JHS, CHS, and KK in 2017. Implementation facilitators included leadership engagement, the previous adoption of the underlying clinical guideline in the nursery, a positive implementation climate, a strong culture of quality improvement, and newborn nursery educational infrastructure supporting guideline implementation. The resulting implementation strategy had three parts: 1) an email communication to end users of the EHR by the institutional IT leadership with information on the innovation, 2) socialization among attending providers through clinical champions, and 3) attending physicians and senior residents teaching incoming residents to use the innovation.

Implementation outcomes assessment started with beta users (JHS, CHS, and other attending physicians added gradually) who provided continuous feedback to help improve innovation usability and accuracy through weekly meetings. The Neonatal Bilirubin Management App was rolled out in the nursery and 12 community clinics in 2017. Implementation outcomes included innovation adoption, reach and implementation fidelity and were assessed by PVK in 2019. According to analysis of the app logs, in 2018, innovation was used at least once (adoption) in the nursery and 12 outpatient clinics. Innovation reach extended to all newborns born at the UU Health because the app was available system-wide. Most uses (86%) occurred in the nursery with physicians, nurses, as well as a variety of other stakeholders including medical students and medical assistants using the innovation. 27 Innovation implementation fidelity was high, with bilirubin levels clinically managed via the innovation for 90% of babies born at UU Health. We continue to monitor reach and implementation fidelity through the weekly analysis of the EHR logs.

Step 8. Initial user satisfaction assessment

PVK, HSK, TT, and CW evaluated initial user satisfaction through qualitative analysis of 12 physician interviews. User response to the intervention was positive and some unintentional uses were uncovered such as using the app for resident education.

Step 9. Technical implementation assessment

Technical implementation assessment involved performance, scalability, and uptime analysis. It was performed by the software development team.

IV. Operation phase

Step 10. technical portability assessment.

Technical portability was evaluated through review of FHIR APIs that were not yet universally supported through US Core FHIR APIs performed by PVK, KK, CN, and the software development team, including through a formal analysis published in 2019. 48

Step 11. Long-term user satisfaction assessment

PVK, TT, and CW evaluated long-term user satisfaction through quantitative analysis of 109 SUS surveys of 208 recent innovation end users identified through system logs (52% response rate). The mean SUS score for attending providers was 91.05 (95% CI, 86.31-95.79) which corresponds to “best imaginable” usability. 47

Step 12. Social outcomes assessment

Social outcomes assessment involved process, health, and economic outcomes analysis by PVK. Process impact assessment was conducted as a before-and-after retrospective clinical trial from April 1, 2016 to April 30, 2019. With regard to process outcomes, this study showed that orders for clinically appropriate phototherapy during hospitalization increased for newborns with bilirubin levels above the guideline-recommended threshold (odds ratio, 1.84; 95% CI, 1.16-2.90; P = .01). 47 In the interviews with 12 users conducted by PVK and HSK, we identified that users were using the app for teaching. No negative unintended process consequences were found. We did not find a significant difference in measured health outcomes including length of stay, intensive care unit admissions, and readmissions. 47 No negative unintended health consequences were found. A preliminary economic evaluation was conducted in 2019. If deployed through the entire United States, given over 3.4 million eligible births in 2018, 66 seconds saved per use, and 5 uses per newborn, the innovation could save over 300,000 hours of clinician time annually, 47 which for a $50-100 hourly pay would translate to $15-30 million in savings. We are exploring the conduct of a more formal cost-effectiveness analysis in the future.

In this paper, we described the ELICIT framework, an evaluation framework that can be used to support all phases in the EHR-integrated innovation life cycle through a comprehensive, replicable process. ELICIT framework provides potential evaluation questions and methods for each measure level (society, user, and IT). ELICIT framework incorporates constructs and methods from health informatics and three related domains: social sciences, human factors engineering, and software engineering. Leveraging these measures, ELICIT framework builds on the EPIS implementation framework, with consideration given to outer and inner implementation contexts, bridging factors, intervention characteristics, and intervention adaptations after implementation. 7 , 32 , 53 As a holistic evaluation framework, ELICIT framework may enable shared understanding and enhanced coordination among stakeholders – a necessary foundation for good team function. Moreover, it may promote evaluation continuity across the IT life cycle by explicitly linking the findings from the initial steps of gathering requirements ( Table 2 ), to designing and developing the software ( Table 3 ), to clinical implementation ( Table 4 ), and to evaluating implementation, effectiveness, and economic outcomes ( Table 5 ). ELICIT framework also supports moving back to earlier phases as needed, which is important in the healthcare domain as both EHR-integrated innovations and their implementation contexts are constantly co-evolving. Evaluators do not need to complete all 12 steps and should prioritize the ones which matter the most to them based on the detailed questions provided in Tables 2 - ​ -5 5 .

Operation Phase: Evaluation Steps, Exemplar Questions, and Exemplar Methods

CMS – Centers for Medicare & Medicaid Services, EHR – electronic health record.

The ELICIT framework was specifically designed to support the evaluation of EHR-integrated innovations. ELICIT’s IT-level measures address issues unique to EHR integration, including interoperability and portability across EHR platforms – a pervasive problem. Similarly, ELICIT’s user-level measures address the challenges associated with integrating EHR-based innovations into existing EHR user interfaces and clinical workflows. Furthermore, ELICIT’s society-level measures take into consideration how the EHR-based innovations affect organizational culture, business processes, patient outcomes and finances. These society-level measures are important for catalyzing further investments in, and wide dissemination of, EHR-integrated innovations.

Compared to other health informatics evaluation frameworks, the ELICIT framework adds 1) coverage of the entire IT life cycle, 2) description of measures specific to society, user, and IT levels, 3) measures specific to EHR-integrated innovations, 4) attention to clinical context perspectives, 5) attention to implementation science, and 6) a checklist of relevant questions and methods that could be considered at each phase. The ELICIT framework is also more concrete than existing frameworks because it provides a checklist of relevant exemplar questions and methods that could be considered at each phase. These concrete questions and methods can serve as a starting point for evaluation activities. While some aspects of the framework are similar to existing frameworks, none of the existing frameworks provide a matrix which covers every intersection between the four IT life cycle phases and the three levels of measures (i.e., society, user, and IT). For example, ELICIT framework includes all the constructs defined in HITREF: 18 , 19 structural quality and quality of information logistics are covered in steps of the IT level; unintended consequences/benefits, effects on quality process, and barrier and facilitators to adoption are covered in the society level. Moreover, ELICIT framework incorporates elements from human factors engineering, software engineering, and implementation science which are not included in HITREF. In addition, we believe ELICIT framework is easier to follow because evaluation steps are assigned to discrete IT life cycle phases.

To our knowledge, this is the first evaluation framework for EHR-integrated innovations that focuses on appropriate evaluation processes throughout the full IT life cycle that specifically includes the implementation phase, a much neglected aspect of information technology projects. 72 , 107 Many evaluation frameworks focus on the evaluation of only one part of the IT life cycle such as human factors engineering. 22 , 23 Even evaluation frameworks that focus on full-cycle approaches do not integrate all the different steps, especially the last few steps of clinical implementation. 5 , 26 - 29 Yet, effective implementation strategies are likely to be as important for overall innovation success as design quality or clinical value. Thus, although existing innovation evaluation frameworks have commonalities with ELICIT framework such as a focus on accuracy, usability, safety and effectiveness, 5 , 26 - 29 the ELICIT framework adds to the existing research by explicitly focusing on clinical implementation strategies and outcomes.

Researchers in the field of implementation science also speak to the need to engage stakeholders, organize resources, and address user needs. 7 , 31 , 32 , 53 , 65 , 66 However, they tend to underemphasize the IT aspects, EHR workflow, communication channels, and user mental models that are required for gathering functionality requirements, designing system components, and designing effective implementation strategies. 34 Thus, ELICIT framework adds to existing implementation science research as it focuses on software engineering, user requirements, human factors engineering and value assessment.

Implications

Subject to further empirical testing, ELICIT framework could be applied to evaluation facilitation, coordination, and budgeting. First and foremost, we believe that ELICIT framework can facilitate effective, holistic, scalable evaluations of EHR-integrated innovations, thereby increasing their collective capacity to improve patient care, the provider experience and healthcare value. Second, ELICIT framework can facilitate planning and coordination for the evaluation of EHR-integrated innovations by identifying the evaluation needs across the IT life cycle. Third, we believe ELICIT framework can help justify the investment in evaluation efforts by explicitly mapping out the required evaluation steps and by supporting successful dissemination. Therefore, we hope that this framework will help innovative healthcare organizations and health IT companies to establish evaluation programs, select and pursue research questions and methods, and report the results of their development and evaluation efforts.

We believe that this framework could potentially be generalized to all health IT innovations including EHRs, consumer-facing mobile apps, and artificial intelligence, and we encourage others to adapt and test our framework in different settings. We are planning to continue evaluating and validating this framework in the future. If the ELICIT framework is successfully validated, it would suggest that our methodology for framework development based on combining literature review, expert consensus, and practical experience might also be generalizable.

Limitations

A limitation of this study is that ELICIT framework was developed predominantly at one institution and has not been tested by external investigators. However, this framework was developed based on our experiences with a wide range of projects including operationally and grant funded projects in a wide range of clinical domains and settings. 41 - 48 , 50 We also collaborated with investigators from many other organizations and are familiar with their evaluation needs and approaches. As a second limitation, it is possible that some aspects of our evaluation framework may be difficult to implement in settings with less access to relevant experts. Finally, this is a work still in progress. As such, further enhancements may arise based on the experience of ourselves and others.

While EHR-integrated innovations show great promise, inadequate evaluations may result in poor usability, adoption, safety, or impact. A 12-step multi-level multi-phase evaluation framework for EHR-integrated innovations was iteratively developed to integrate insights from relevant domains and to help bring impactful innovations to market in an efficient and safe manner. This comprehensive and integrated framework covers the entire IT life cycle across the society, user and IT-measures, whereas previous frameworks focused on specific aspects only. We hope that this work will support evaluations undertaken by other digital health innovation teams while fostering discussion and further refinement of the framework.

Box 1. Definitions

Adaptation is “a process of thoughtful and deliberate alteration to the design or delivery of an intervention, with the goal of improving its fit or effectiveness in a given context.” 30

Context is the circumstances that form the setting for the innovation and enable it to be fully understood and assessed. 31 Innovation context includes individual factors (e.g., individuals’ knowledge, beliefs, values, goals), organizational factors (e.g., organizational culture, climate and priorities), external factors (e.g., vendors, government regulations, competition, external funding, patient advocacy organizations, public opinion), and technical factors (e.g., system constraints and capabilities, implementation of interoperability standards in the organization).

Implementation is the rollout of innovations into clinical practice. Of note, this definition is different than how the term is used in software engineering, where the term often also encompasses the software coding process.

Information technology (IT) is a technology used to acquire, store, deliver and/or analyze data.

End user is the intended individual who uses the software. For provider-facing innovations, end users may include physicians, pharmacists, nurses, advanced care providers, and medical assistants. For patient-facing innovations, end users are patients and caregivers.

Evidence-based practice (EBP) is an evidence-based intervention, treatment, innovation, recommendation or guideline. 32

Formative evaluation is the evaluation of an evolving innovation to give direction for the design, development, and implementation of innovation components. 33

Implementation strategies are the actions taken to enhance the adoption, clinical rollout, and/or sustainability of innovations. 34

Stakeholders are the individuals and groups affected by the innovation, including software developers, evaluators, end users, and organizational leaders.

Summative evaluation is describing the properties and/or impacts of the innovation. 33

• It is critical for EHR-integrated innovations to be systematically evaluated
• Effective evaluation of EHR-integrated innovations requires a shared understanding and collaboration across disciplines throughout the full information technology (IT) life cycle
• A novel Evaluation in Life Cycle of Information Technology (ELICIT) framework focuses on all phases of the IT life cycle
• The ELICIT framework consists of 12 evaluation steps across the IT life cycle phases including society-, user-, and IT-level measures

ACKNOWLEDGMENTS

We would like to thank the ReImagine EHR software development team, and especially Dr. Salvador Rodriguez-Loya, for their contribution to the development of EHR-integrated innovations. We would also like to thank the UU Print and Mail Services team for their contribution to the evaluation framework visualization.

FUNDING STATEMENT

The work reported in this paper was supported in part by the University of Utah, the Agency for Healthcare Research and Quality under Award Number R18HS026198. TJR and KLM were supported by the U.S. National Library of Medicine of the National Institutes of Health through grant T15LM007124.

The funding organizations had no role in the conceptualization, design, data collection, analysis, decision to publish, or paper preparation for this case study. The content is solely the responsibility of the authors and does not necessarily represent the official views of the organizations involved.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

COMPETING INTERESTS STATEMENT

KK reports honoraria, consulting, sponsored research, licensing, or co-development with McKesson InterQual, Hitachi, Pfizer, the Korean Society of Medical Informatics, NORC at the University of Chicago, Premier, Klesis Healthcare, RTI International, Mayo Clinic, Vanderbilt University, the University of Washington, Indiana University, the University of California at San Francisco, MD Aware, and the U.S. Office of the National Coordinator for Health IT (via ESAC, JBS International, A+ Government Solutions, Hausam Consulting, and Security Risk Solutions) in the area of health information technology. KK was also an unpaid board member of the non-profit Health Level Seven International health IT standard development organization, and he is an unpaid member of the U.S. Health Information Technology Advisory Committee. CJS reports sponsored research with Hitachi, Ltd. Other co-authors have no conflicts to report.

Contributor Information

Polina V. Kukhareva, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.

Charlene Weir, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.

Guilherme Del Fiol, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.

Gregory A. Aarons, Department of Psychiatry, UC San Diego ACTRI Dissemination and Implementation Science Center, UC San Diego, La Jolla, CA, USA.

Teresa Y. Taft, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.

Chelsey R. Schlechter, Department of Population Health Sciences, Center for Health Outcomes and Population Equity, Huntsman Cancer Institute, University of Utah, Salt Lake City, UT, USA.

Thomas J. Reese, Department of Biomedical Informatics, Vanderbilt University, Nashville, TN, USA.

Rebecca L. Curran, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.

Claude Nanjo, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.

Damian Borbolla, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.

Catherine J Staes, College of Nursing, University of Utah, Salt Lake City, UT, USA.

Keaton L. Morgan, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.

Heidi S. Kramer, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.

Carole H. Stipelman, Department of Pediatrics, University of Utah, Salt Lake City, UT, USA.

Julie H. Shakib, Department of Pediatrics, University of Utah, Salt Lake City, UT, USA.

Michael C. Flynn, Department of Family & Preventive Medicine, University of Utah, Salt Lake City, UT, USA.

Kensaku Kawamoto, Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.