research paper on scrum methodology

Risks and Challenges of Scrum: A Systematic Literature Review

• First Online: 25 August 2022

Cite this chapter

• Pedro Filipe Soares   ORCID: orcid.org/0000-0001-5460-0886 5  

Part of the book series: Studies in Systems, Decision and Control ((SSDC,volume 210))

841 Accesses

1 Citations

Agile methodologies were increasingly adopted in the last decades. In the Agile methodologies’ universe, Scrum is the most used one and is gaining broader acceptance. In fast changing times, marked by various uncertainties in the organization of workplaces, location of teams and technological challenges, it is necessary to update the risks and challenges of using Scrum. With updated information and awareness of the risks and challenges inherent to the SCRUM methodology, the decision on whether to adopt this methodology will be better addressed. The research is made in the form of a systematic literature review (SLR). This SLR extracted data from 39 papers published from 2017 to 2021, a period that encompasses important impacts of the ongoing transformations with the adoption of cloud computing, the use of big data and the difficulties that the COVID-19 pandemic posed around the world. Several challenges and risks were identified, from team communication to collaboration with stakeholders, from effort estimation to knowledge management, and suggestions are made to improve Scrum adoption success rate.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Benefits and Challenges of Adopting the Scaled Agile Framework (SAFe): Preliminary Results from a Multivocal Literature Review

Agile Challenges in Practice: A Thematic Analysis

Evolution of Scrum Transcending Business Domains and the Future of Agile Project Management

https://www.scimagojr.com .

Adnan, M., Afzal, M.: Ontology based multiagent effort estimation system for scrum agile method. IEEE Access 5 , 25993–26005 (2017). https://doi.org/10.1109/ACCESS.2017.2771257

Article   Google Scholar  

Al-Zaidi, A., Qureshi, R.: Global software development geographical distance communication challenges. Int. Arab J. Inf. Technol. 14 (2) (2017). https://www.umc.edu.dz/images/Global-Software-Development-Geographical-Distance-Communication-Challenges.pdf

Amorim, A.C., Mira da Silva, M., Pereira, R., Gonçalves, M.: Using agile methodologies for adopting COBIT. Inf. Syst. 101 , 101496 (2021). https://doi.org/10.1016/j.is.2020.101496

Baham, C.: Improving business product owner commitment in student scrum projects. J. Inf. Technol. Educ.: Res. 19 , 243–258 (2020). https://doi.org/10.28945/4549

Beck, K., Beedle, M., Bennekum, A. van, Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R.C., Mellor, S., Schwaber, K., Sutherland, J., & Thomas, D.: The Agile Manifesto (2001). http://agilemanifesto.org/

Bick, S., Spohrer, K., Hoda, R., Scheerer, A., Heinzl, A.: Coordination challenges in large-scale software development: a case study of planning misalignment in hybrid settings. IEEE Trans. Softw. Eng. 44 (10), 932–950 (2018). https://doi.org/10.1109/TSE.2017.2730870

Bourdeau, S., Coulon, T., Petit, M.-C.: Simulation-based training via a “readymade” virtual world platform: teaching and learning with minecraft education. IT Prof. 23 (2), 33–39 (2021). https://doi.org/10.1109/MITP.2021.3062935

Burch, M., Melby, E.: What more than a hundred project groups reveal about teaching visualization. J. Vis. 23 (5), 895–911 (2020). https://doi.org/10.1007/s12650-020-00659-6

Conboy, K., Carroll, N.: Implementing large-scale agile frameworks: challenges and recommendations. IEEE Softw. 36 (2), 44–50 (2019). https://doi.org/10.1109/MS.2018.2884865

Cornide-Reyes, H., Noël, R., Riquelme, F., Gajardo, M., Cechinel, C., mac Lean, R., Becerra, C., Villarroel, R., Munoz, R.: introducing low-cost sensors into the classroom settings: improving the assessment in agile practices with multimodal learning analytics. Sensors 19 (15), 3291 (2019). https://doi.org/10.3390/s19153291

de Beurs, D., van Bruinessen, I., Noordman, J., Friele, R., van Dulmen, S.: Active involvement of end users when developing web-based mental health interventions. Front. Psych. 8 (2017). https://doi.org/10.3389/fpsyt.2017.00072

Dingsøyr, T., Nerur, S., Balijepally, V., Moe, N.B.: A decade of agile methodologies: towards explaining agile software development. J. Syst. Softw. 85 (6), 1213–1221 (2012). https://doi.org/10.1016/j.jss.2012.02.033

Edison, H., Wang, X., Conboy, K.: Comparing methods for large-scale agile software development: a systematic literature review. IEEE Trans. Softw. Eng. 1–1 (2021). https://doi.org/10.1109/TSE.2021.3069039

Fernandez-Diego, M., Mendez, E.R., Gonzalez-Ladron-De-Guevara, F., Abrahao, S., Insfran, E.: An update on effort estimation in agile software development: a systematic literature review. IEEE Access 8 , 166768–166800 (2020). https://doi.org/10.1109/ACCESS.2020.3021664

Greening, D.R.: Enterprise Scrum: scaling Scrum to the executive level. In: 2010 43rd Hawaii International Conference on System Sciences, pp. 1–10 (2010). https://doi.org/10.1109/HICSS.2010.186

Hadi, S.H., Permanasari, A.E., Hartanto, R., Sakkinah, I.S., Sholihin, M., Sari, R.C., Haniffa, R.: Developing augmented reality-based learning media and users’ intention to use it for teaching accounting ethics. Educ. Inf. Technol. (2021). https://doi.org/10.1007/s10639-021-10531-1

Hamid, M., Zeshan, F., Ahmad, A., Ahmad, F., Hamza, M.A., Khan, Z.A., Munawar, S., Aljuaid, H.: An Intelligent recommender and decision support system (IRDSS) for effective management of software projects. IEEE Access 8 , 140752–140766 (2020). https://doi.org/10.1109/ACCESS.2020.3010968

Hendler, S.: Exploring coordination practices in digital–physical product development. J. Manuf. Technol. Manag. 32 (3), 742–771 (2020). https://doi.org/10.1108/JMTM-06-2019-0229

Herrera, F., Niño, R., Montenegro-Marín, C.E., Gaona-García, P.A., de Mendívil, I.S.M., Crespo, R.G.: Computational method for monitoring pauses exercises in office workers through a vision model. J. Ambient. Intell. Humaniz. Comput. 12 (3), 3389–3397 (2021). https://doi.org/10.1007/s12652-020-02391-3

Hoda, R., Salleh, N., Grundy, J., Tee, H.M.: Systematic literature reviews in agile software development: a tertiary study. Inf. Softw. Technol. 85 , 60–70 (2017). https://doi.org/10.1016/j.infsof.2017.01.007

Hummel, M., Epp, A.: Success factors of agile information systems development: a qualitative study. In: 2015 48th Hawaii International Conference on System Sciences, pp. 5045–5054 (2015). https://doi.org/10.1109/HICSS.2015.598

Kiani, A.A., Hafeez, Y., Imran, M., Ali, S.: A dynamic variability management approach working with agile product line engineering practices for reusing features. J. Supercomput. 77 (8), 8391–8432 (2021). https://doi.org/10.1007/s11227-021-03627-5

Kitchenham, B.: Procedures for Performing Systematic Reviews. Keele University (2004). https://www.inf.ufsc.br/~aldo.vw/kitchenham.pdf

Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., Linkman, S.: Systematic literature reviews in software engineering—a systematic literature review. Inf. Softw. Technol. 51 (1), 7–15 (2009). https://doi.org/10.1016/j.infsof.2008.09.009

Larman, C., Basili, V.R.: Iterative and incremental developments. A brief history. Computer 36 (6), 47–56 (2003). https://doi.org/10.1109/MC.2003.1204375

Lei, H., Ganjeizadeh, F., Jayachandran, P.K., Ozcan, P.: A statistical analysis of the effects of Scrum and Kanban on software development projects. Robot. Comput. Integr. Manuf. 43 , 59–67 (2017). https://doi.org/10.1016/j.rcim.2015.12.001

Levy, Y., Ellis, J.T.: A systems approach to conduct an effective literature review in support of information systems research. Inf. Sci. Int. J. Emerg. Transdiscipl. 9 , 181–212 (2006). https://doi.org/10.28945/479

Marques, R., Costa, G., Mira da Silva, M., Gonçalves, D., Gonçalves, P.: A gamification solution for improving Scrum adoption. Empir. Softw. Eng. 25 (4), 2583–2629 (2020). https://doi.org/10.1007/s10664-020-09816-9

Martinez, D., Ferre, X., Guerrero, G., Juristo, N.: An agile-based integrated framework for mobile application development considering ilities. IEEE Access 8 , 72461–72470 (2020). https://doi.org/10.1109/ACCESS.2020.2987882

Masood, Z., Hoda, R., Blincoe, K.: Adapting agile practices in university contexts. J. Syst. Softw. 144 , 501–510 (2018). https://doi.org/10.1016/j.jss.2018.07.011

Mathrani, A., Wickramasinghe, S., Jayamaha, N.P.: An evaluation of documentation requirements for ISO 9001 compliance in scrum projects. TQM J. (ahead-of-print) (2021). https://doi.org/10.1108/TQM-08-2020-0177

Mollahoseini Ardakani, M.R., Hashemi, S.M., Razzazi, M.: Adapting the Scrum methodology for establishing the dynamic inter-organizational collaboration. J. Organ. Chang. Manag. 31 (4), 852–866 (2018). https://doi.org/10.1108/JOCM-07-2016-0135

Mollahoseini Ardakani, M.R., Hashemi, S.M., Razzazi, M.: A cloud-based solution/reference architecture for establishing collaborative networked organizations. J. Intell. Manuf. 30 (5), 2273–2289 (2019). https://doi.org/10.1007/s10845-017-1387-2

Moniruzzaman, A.B.M., Hossain, D.S.A.: Comparative study on agile software development methodologies. Glob. J. Comput. Sci. Technol. 13 (7) (2013)

Google Scholar  

Mora, M., Adelakun, O., Galvan-Cruz, S., Wang, F.: Impacts of IDEF0-based models on the usefulness, learning, and value metrics of Scrum and XP project management guides. Eng. Manag. J. 1–17 (2021). https://doi.org/10.1080/10429247.2021.1958631

Noordman, J., Driesenaar, J.A., van Bruinessen, I.R., van Dulmen, S.: Listening time; participatory development of a web-based preparatory communication tool for elderly cancer patients and their healthcare providers. Internet Interv. 9 , 51–56 (2017). https://doi.org/10.1016/j.invent.2017.05.002

Okoli, C.: A guide to conducting a standalone systematic literature review. Commun. Assoc. Inf. Syst. 37 (2015). https://doi.org/10.17705/1CAIS.03743

Özcan-Top, Ö., McCaffery, F.: To what extent the medical device software regulations can be achieved with agile software development methods? XP—DSDM—Scrum. J. Supercomput. 75 (8), 5227–5260 (2019). https://doi.org/10.1007/s11227-019-02793-x

Pócsová, J., Bednárová, D., Bogdanovská, G., Mojžišová, A.: Implementation of agile methodologies in an engineering course. Educ. Sci. 10 (11), 333 (2020). https://doi.org/10.3390/educsci10110333

Przybyłek, A., Albecka, M., Springer, O., Kowalski, W.: Game-based Sprint retrospectives: multiple action research. Empir. Softw. Eng. 27 (1), 1 (2022). https://doi.org/10.1007/s10664-021-10043-z

Saeedi, K., Visvizi, A.: Software development methodologies, HEIs, and the digital economy. Educ. Sci. 11 (2), 73 (2021). https://doi.org/10.3390/educsci11020073

Schwaber, K., Beedle, M.: Agile Software Development with Scrum, vol. 1. Prentice Hall, Upper Saddle River (2002)

Scott, E., Campo, M.: An adaptive 3D virtual learning environment for training software developers in scrum. Interact. Learn. Environ. 1–19 (2021). https://doi.org/10.1080/10494820.2021.1999985

Shafiee, S., Wautelet, Y., Hvam, L., Sandrin, E., Forza, C.: Scrum versus rational unified process in facing the main challenges of product configuration systems development. J. Syst. Softw. 170 , 110732 (2020). https://doi.org/10.1016/j.jss.2020.110732

Sievi-Korte, O., Richardson, I., Beecham, S.: Software architecture design in global software development: an empirical study. J. Syst. Softw. 158 , 110400 (2019). https://doi.org/10.1016/j.jss.2019.110400

Spiegler, S.V., Heinecke, C., Wagner, S.: An empirical study on changing leadership in agile teams. Empir. Softw. Eng. 26 (3), 41 (2021). https://doi.org/10.1007/s10664-021-09949-5

Steghöfer, J.-P., Burden, H., Alahyari, H., Haneberg, D.: No silver brick: opportunities and limitations of teaching Scrum with Lego workshops. J. Syst. Softw. 131 , 230–247 (2017). https://doi.org/10.1016/j.jss.2017.06.019

Sutherland, J.: Future of Scrum: parallel pipelining of sprints in complex projects. In: Agile Development Conference (ADC’05), pp. 90–99 (2005). https://doi.org/10.1109/ADC.2005.28

Takeuchi, H., Nonaka, I.: The new product development game. J. Prod. Innov. Manag. 3 (3), 205–206 (1986). https://doi.org/10.1016/0737-6782(86)90053-6

Tavares, B.G., da Silva, C.E.S., de Souza, A.D.: Risk management analysis in Scrum software projects. Int. Trans. Oper. Res. 26 (5), 1884–1905 (2019). https://doi.org/10.1111/itor.12401

Urbieta, M., Antonelli, L., Rossi, G., do Prado Leite, J.C.S.: The impact of using a domain language for an agile requirements management. Inf. Softw. Technol. 127 , 106375 (2020). https://doi.org/10.1016/j.infsof.2020.106375

Vallon, R., da Silva Estácio, B.J., Prikladnicki, R., Grechenig, T.: Systematic literature review on agile practices in global software development. Inf. Softw. Technol. 96 , 161–180 (2018). https://doi.org/10.1016/j.infsof.2017.12.004

Download references

Author information

Authors and affiliations.

Instituto Universitário de Lisboa (ISCTE-IUL), Lisbon, Portugal

Pedro Filipe Soares

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Pedro Filipe Soares .

Editor information

Editors and affiliations.

Instituto Universitário de Lisboa (ISCTE-IUL), Lisboa, Portugal

Ruben Pereira

Universidade Federal de Santa Catarina, Santa Catarina, Brazil

Isaias Bianchi

ISEG, University of Lisbon, Lisbon, Portugal

Álvaro Rocha

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Soares, P.F. (2022). Risks and Challenges of Scrum: A Systematic Literature Review. In: Pereira, R., Bianchi, I., Rocha, Á. (eds) Digital Technologies and Transformation in Business, Industry and Organizations. Studies in Systems, Decision and Control, vol 210. Springer, Cham. https://doi.org/10.1007/978-3-031-07626-8_9

Download citation

DOI : https://doi.org/10.1007/978-3-031-07626-8_9

Published : 25 August 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-07625-1

Online ISBN : 978-3-031-07626-8

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER COLUMN
• 10 April 2019

How agile project management can work for your research

• Laura Pirro 0

Laura Pirro is a PhD student in chemical engineering at the Laboratory for Chemical Technology at Ghent University, Belgium.

You can also search for this author in PubMed   Google Scholar

If you’ve ever written a research proposal, the chances are that you will have planned the work as a list of sequential activities, often visualized in a Gantt chart.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-019-01184-9

This is an article from the Nature Careers Community, a place for Nature readers to share their professional experiences and advice. Guest posts are encouraged. You can get in touch with the editor at [email protected].

Sutherland, J. Scrum: The Art of Doing Twice the Work in Half the Time (Random House Business, 2015).

Google Scholar  

Stark, E. Agile Project Management: Quick Start Guide (ClydeBank Media, 2014).

Download references

Related Articles

How to run a creative and diverse PhD programme

• Research management

US halts funding to controversial virus-hunting group: what researchers think

News 16 MAY 24

Japan can embrace open science — but flexible approaches are key

Correspondence 07 MAY 24

US funders to tighten oversight of controversial ‘gain of function’ research

News 07 MAY 24

I’m worried I’ve been contacted by a predatory publisher — how do I find out?

Career Feature 15 MAY 24

How I fled bombed Aleppo to continue my career in science

Career Feature 08 MAY 24

Illuminating ‘the ugly side of science’: fresh incentives for reporting negative results

Reading between the lines: application essays predict university success

Research Highlight 17 MAY 24

How to stop students cramming for exams? Send them to sea

News & Views 30 APR 24

How young people benefit from Swiss apprenticeships

Spotlight 17 APR 24

Postdoc in CRISPR Meta-Analytics and AI for Therapeutic Target Discovery and Priotisation (OT Grant)

APPLICATION CLOSING DATE: 14/06/2024 Human Technopole (HT) is a new interdisciplinary life science research institute created and supported by the...

Human Technopole

Research Associate - Metabolism

Houston, Texas (US)

Baylor College of Medicine (BCM)

Postdoc Fellowships

Train with world-renowned cancer researchers at NIH? Consider joining the Center for Cancer Research (CCR) at the National Cancer Institute

Bethesda, Maryland

NIH National Cancer Institute (NCI)

Faculty Recruitment, Westlake University School of Medicine

Faculty positions are open at four distinct ranks: Assistant Professor, Associate Professor, Full Professor, and Chair Professor.

Hangzhou, Zhejiang, China

Westlake University

PhD/master's Candidate

PhD/master's Candidate    Graduate School of Frontier Science Initiative, Kanazawa University is seeking candidates for PhD and master's students i...

Kanazawa University

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.5(3); 2019 Mar

Adapting the scrum framework for agile project management in science: case study of a distributed research initiative

This article explores the adoption of agile methods for the management of projects in collaborative research initiatives. The use of the scrum framework, a specific set of agile principles and practices for self-organizing cross-functional teams in software development projects, is currently being expanded to other types of organizations and knowledge management processes. The study addresses the extent to which key principles and tools usually used in scrum, due to their potentially positive influence on team dynamics and efficiency, can contribute to the collaborative management and coordination of tasks in research processes. The responses from interviews with 17 researchers, as well as participant observation and analysis of online activity, are examined and presented as a case study on the adoption of scrum practices in a distributed research centre dedicated to the evaluation of public policies. Results indicate that integrating agile methods and principles for interdisciplinary collaboration requires a high degree of flexibility and a “learn by doing” approach.

1. Introduction

1.1. team-based collaboration in research.

Team-based collaboration is a critical factor in research organizations and scientific fields, as knowledge is increasingly being generated by research teams ( Wuchty et al., 2007 ; Wagner et al., 2017 ). Literature on research practices indicates that teamwork and collaboration dominate knowledge production in academic organizations and is prevalent in large-scale international research networks ( Cooke and Hilton, 2015 ). Academics and investigatory teams working on science, engineering and social science disciplines have shifted towards collective research ( Wuchty et al., 2007 ). The benefits of research collaboration range from an increase in citations as a result of the co-authorship of papers to better use of existing resources ( Ynalvez and Shrum, 2011 ). Other benefits include the capacity to generate wider social impact through large-scale research projects ( Bammer, 2008 ), and more opportunities for knowledge transfer and learning ( Lassi and Sonnenwald, 2010 ) or for managing complexity ( Helbing et al., 2015 ).

The study of collaborative research networks from diverse perspectives has gained momentum in recent years ( Wang and Hicks, 2015 ) because funding agencies, which prioritise better use of existing resources, prestige and international reputation, are encouraging large-scale collaborative research programs ( Smykla and Zippel, 2010 ). In this respect, research collaboration may be viewed as a self-assembling entity, characterized by fuzzy boundaries and the tendency to function as networks ( Spinuzzi, 2015 ; Wang and Hicks, 2015 ) that involve not only different research institutions, but also expand to include collaboration with industry, governments or civil society ( Bridgeford and Amant, 2017 ). Collaboration may occur across sectors and types of organisations ( Bozeman and Corley, 2004 ), such as government-based research programs, that usually emphasize multidisciplinary and applied research ( Gray et al., 2001 ), or in industry, where the confines of conducting research are usually bypassed for the sake of academic publishing and the search for utility for the non-academic partners ( Perkmann and Walsh, 2009 ).

Several authors contend that this shift to research collaboration is occurring amidst a trend towards disruptive adoption of information and communication technologies (ICT) in knowledge-intensive organizations ( Jirotka et al., 2013 ; Borgman, 2010 ; Powell and Snellman, 2004 ). At present, collective research is undertaken in more distributed, reflexive and less hierarchical work arrangements ( Zuboff, 1988 ), thereby expanding the possibilities for complex multidisciplinary and interdisciplinary collaborations on varying scales ( König et al., 2013 ). In parallel to the prevailing opinion that research collaboration correlates with high productivity ( Daradoumis et al., 2012 ) and quality results ( Rigby and Edler, 2005 ; Liao, 2011 ), some scholars describe it as a difficult and ever-changing process, particularly when involving collaboration between geographically dispersed remote teams ( Eccles et al., 2009 ). Key challenges in team-based collaborative research management relate to issues of commitment, transparency or communication and monitoring ( Keraminiyage et al., 2009 ). Collaboration across disciplines also requires progressive adaptation of a shared language and different types of tools ( Jeffrey, 2003 ).

Kraut et al. (1987) , describing the process of collaborative research in scientific teams, explain how plans become progressively more detailed and specific, but can often be revised and even abandoned without negatively impacting collaboration. Other challenges in collaborative research management relate to the need for supervision and coordination among peers ( Delfanti, 2016 ), or to coordinating an activity that is continually evolving ( König et al., 2013 ). Large-scale research projects usually imply more dedication to leading and coordinating each process, from research design to the collaborative authorship of papers and reports ( Bozeman and Corley, 2004 ). In this sense, collaborative research projects often require new project management techniques ( Vom Brocke and Lippe, 2015 ). Methodologically, these additional complexities when performing scientific activities represent an evolving interdisciplinary field requiring various types of analysis of how and when collaborative research is implemented ( Sonnenwald, 2007 ; Katz and Martin, 1997 ).

1.2. Agile project management

Agile project management (APM) or “agile methods” represents a team management approach and a productivity framework that supports continuous and incremental progress on work priorities, even in the face of changes. APM has its origins in the agile processes of software development, such as Scrum, XP, DSDM, Cristal, etc., which are programming methodologies based on adaptability to any change as a means to increase the chances of success of a project ( Cohen et al., 2004 ). Most agile methods try to minimize risks during the execution of a project by developing software in iterations, which usually last from one to four weeks. Each iteration is like a miniature project of the final project, and includes all the tasks necessary to implement new functionalities: planning, requirements analysis, design, coding, testing, and documentation. An agile programming project aims to release new software at the end of each iteration, and between each iteration the team reevaluates its priorities.

APM has gained in popularity in recent years, primarily in the software industry ( Scrum Alliance, 2016 ) but is progressively breaking into other domains ( Ciric et al., 2018 ). In the late 1990s software development teams started to apply agile methods for the improvement of programming processes by making them more continuous and incremental on the basis of agile principles such as adaptability, personal and group autonomy, modularity and self-organized collaboration, as defined in the Agile manifesto ( Beck et al., 2001 ). The manifesto was a reaction to the weaknesses and rigidity of popular plan-based software production methodologies, such as the previously highly acclaimed “waterfall” method, which has been criticised mainly for its lack of responsiveness to change ( Cockburn, 2002 ). APM, more so than other management frameworks, emphasizes teamwork by focusing on the social aspects of software development ( Rosenberg and Stephens, 2003 ), channelling co-creation between programmers and other participants in self-organized, cross-functional teams ( Hoda et al., 2013 ), with collective ownership and collective responsibility as key attributes ( Robinson and Sharp, 2005 ). According to Conforto et al. (2014) APM practices include: (1) the use of the “project vision” concept, (2) simple communication tools and processes, (3) iterative planning, (4) developing activities via self-managed and self-directed teams, and (5) frequently applying project plan monitoring and updating activities.

Despite the critique by some authors that the agile manifesto principles are insufficiently grounded in theory ( Conboy and Fitzgerald, 2004 ) and claims that APM practices and principles lack focus on software architecture ( Rosenberg and Stephens, 2003 ), that it is suitable for small teams but not larger projects ( Cohen et al., 2004 ), and that it is not a panacea for effective project management ( Veneziano et al., 2014 ), the majority of peer-reviewed papers and other empirical studies highlight the benefits of adopting agile methods ( Dybå and Dingsøyr, 2008 ). The growing use of APM seems mainly due to the potential for optimizing the operative capacity of teamwork in short implementation cycles and the positive influence exerted on team dynamics ( Fernandez and Fernandez, 2008 ). Some other documented benefits of the adoption of agile methods relate to the visualization and sharing of progress on tasks, thereby maximizing possibilities for success in projects in complex and multidisciplinary environments ( Cao et al., 2009 ).

As indicated earlier in this discussion, the use of APM has expanded beyond software development to other organizational contexts ( Ciric et al., 2018 ; Rigby et al., 2016 ). Analyses have been conducted on the implementation of agile management in product development ( Lehnen et al., 2016 ; Stare, 2014 ), educational projects ( Grimheden, 2013 ), construction projects ( Demir and Theis, 2016 ), venture capital groups ( Sutherland and Altman, 2009 ), innovation processes ( Hannola et al., 2013 ) and the management of projects in libraries ( Niemi-Grundström, 2014 ) and banks (Niclasen and Stoklund, 2016). In parallel to evidence of the contribution of AMP to a more flexible and responsive organizational culture outside of the software development world ( Küpper, 2016 ), there is increasingly more academic literature on the adoption of agile methods for different types of collaborative research processes and scientific projects. For example, studies highlight the successful utilisation of APM in academia-industry collaboration ( Sandberg and Crnkovic, 2017 ; Santos et al., 2016 ; Ota, 2010 ); the application of agile methods to faculty work ( Pope-Ruark, 2017 ) and bridging the gap between research and practice in the management of case studies ( Barroca et al., 2015 ). There is evidence of success in enabling collaboration in working with and mentoring PhD students ( Hicks and Foster, 2010 ); developing prototypes in “Action Design” research projects ( Keijzer-Broers and de Reuver, 2016 ); coordinating a large-scale European research project with distributed teams ( Marchesi et al., 2007 ) and for the production of multidisciplinary research reports ( Senabre Hidalgo, 2018 ). APM can also be successfully used in managing a research and development laboratory ( Lima et al., 2012 ); adopting experimental ethnography approaches in the workplace ( Mara et al., 2013 ); using evidence-based projects for behavioural interventions ( Hekler et al., 2016 ); or adapting lean software development in the biopharmaceutical sector ( DeWit, 2011 ) or in human-centred research practices ( Armstrong et al., 2015 ).

1.3. The scrum framework

The scrum framework is one of the most adapted APM principles and practices ( Lei et al., 2017 ). The scrum methodology facilitates the coordinated activity of programmers who break their work into small tasks that can be completed within fixed duration cycles or “sprints”, tracking progress and re-planning in regular meetings in order to develop products incrementally. The first reference to the term “scrum” appeared in Nonaka and Takeuchi's (1995) “The New New Product Development Game”, where it was defined as a holistic approach to flexible, autonomous and dynamic teamwork with six main characteristics, namely “built-in instability, self-organizing project teams, overlapping development phases, ‘multilearning’, subtle control, and organisational transfer of learning.”

In their study on leading technological companies in Japan and in the United States, via interviews with CEOs and engineers about how they developed successful innovative products, the authors identified those key characteristics and defined them as follows. (1) Built-in instability: when top management offers a project team a wide measure of freedom and also establishes challenging goals. (2) Self-organizing project teams: when groups take initiatives and develop an independent agenda for their work. (3) Overlapping development phases: instead of a sequential approach (where a project goes through several phases in a step-by-step fashion) the overlapping approach emphasizes speed and flexibility, and enhances shared responsibility and cooperation. (4) ‘Multilearning’: when team members engage in a continual process of trial and error, “learning by doing” along two dimensions: across multiple levels (individual, group, and institutional) and across diverse functions. (5) Subtle control: although teams can be largely on their own, management establishes checkpoints to prevent instability, ambiguity and tension, while in parallel there's also control through “peer pressure”. (6) Organisational transfer of learning: participants transfer their learning to others outside the group, creating conditions for new projects, and also by assigning key individuals to subsequent projects. Knowledge is also transmitted through the organization by converting project activities to standard practice.

Given the focus on a team's collective intelligence, the scrum framework usually requires facilitation to improve teamwork and motivation, to clarify who's doing what, to help with conflict resolution techniques, and to ensure that team members contribute ( Rigby et al., 2016 ). Like the rest of the team, the facilitator or “Scrum Master”, who can be an experienced colleague or a professional hired for such purpose, works on a Kanban board, which is used to document the elements, as well as enable the social aspects of tasks ( Sharp et al., 2009 ). The Scrum Master, therefore, performs the role traditionally assumed by a project manager or team leader and, in this case, is responsible for implementing scrum values and practices, as well as removing impediments ( Cervone, 2011 ).

Subjecting each task to “development sprints” (a period of work averaging 14–20 days) is another practice that is directly related to the scrum methodology ( Abrahamsson et al., 2017 ). Sprints, which are iterative cycles where a given project is developed or enhanced to produce new increments, are usually initiated with a planning meeting at which participants agree on a list of tasks to be performed by the end of a specified period. During the sprint, the team meets daily in short meetings called “standups” to track work progress and communicate ( Friess, 2018 ) and, if necessary, resolve issues ( Marcal et al., 2007 ). At the end of the sprint, a review or “retrospective” meeting is held at which the team examines developments that occurred during the sprint ( Marcal et al., 2007 ). Interested stakeholders may also attend this meeting. Another scrum practice that is directly related to the APM framework, in this case derived from Lean production models, involves the small, regular releases of “minimum viable products”, as opposed to final, fully completed and evaluated outputs at the end of long periods ( Münch et al., 2013 ).

Whether following the scrum methodology or more “light” and simple aspects of the APM framework, the adoption of a Kanban board is useful for its practicality and for tracking implementation on a daily basis ( Anderson et al., 2012 ). The literal translation of Kanban, which is of Japanese origin, is “visual” (Kan) “board” (Ban). Using Kanban, work is broken down into tasks, with descriptions shown on cards or Post-It notes that are displayed on a shared board (usually with separate columns to reflect process). In this way, workflows are visible to all members of the team ( Ahmad et al., 2013 ). Whether via physical or digital tools, the Kanban board infuses the agile development process with high visibility –providing a means of displaying the work assignments of the team, communicating priorities, making it easier to highlight bottlenecks, and helping to optimize efforts ( Cocco et al., 2011 ). This key aspect of shared visibility and dynamism in the coordination of teamwork —a paradigm focused on doable and transparent tasks— is a basic tenet of the adoption of scrum practices in collaborative processes and organizational structures outside of the software development context ( West et al., 2010 ).

2. Background

As the previous section argues, agile methods constitute an increasingly popular management process based on principles of adaptive planning, continuous improvement, frequent consultation with participants and small and regular releases ( Cao et al., 2009 ), as well as simplicity and dynamism ( Abrahamsson et al., 2017 ). In this paper —an exploratory analysis— the focus is on the appropriation of scrum as a methodological framework and its experimental use in the management of distributed and interdisciplinary research initiatives, with the aim of identifying the experiences and perceptions of researchers in the adoption of APM principles and practices, as well as the potential benefits and limitations.

In this regard, the paper seeks to answer the following research questions:

• • Which conditions favour the appropriation of APM for research collaboration?
• • To what extent can specific scrum principles and tools be adopted in interdisciplinary contexts?
• • What are the limitations and advantages of adapting agile methods in a distributed research organisation?

The UK-based Centre for the Evaluation of Complexity Across the Nexus (CECAN, cecan.ac.uk) is the focus of this case study. CECAN, a research centre hosted by the University of Surrey, was established in 2016 and comprises more than 50 members working in 14 different academic organisations such as the University of Warwick, the University of York, Cranfield University and Newcastle University. Conceived as a network of social scientists, policy makers, policy analysts and experts, CECAN explores, tests and promotes innovative policy evaluation approaches and methods pertaining to food, energy, water and the environment across nexus domains. The organisation carries out this mission through the implementation of a series of ‘real-life’ case study projects with UK partner institutions including the Economic and Social Research Council (ESRC), the Natural Environment Research Council (NERC), the Department for Environment Food and Rural Affairs (DEFRA), and the Department for Business, Energy and Industrial Strategy (BEIS), among others.

CECAN teams develop case studies and other interdisciplinary initiatives around research methodologies, complex systems, policy evaluation (in areas related to sustainability or economic promotion), as well as new evaluation and assessment methods. As a distributed initiative incorporating experts from diverse knowledge areas with varying levels of dedication and time capacity for projects, and in the absence of a central physical office or shared space, it required a specific approach to coordination and management. For this purpose, from its early operations, CECAN adopted some APM principles and practices derived from the scrum framework, as well as a digital Kanban board for managing the information and knowledge generated by its teams.

This case study utilised three methodological approaches and data sources: participant observation, analysis of online activity and semi-structured interviews. This combination of approaches forms the basis for the analysis of the adoption of agile principles and practices and the scrum framework at CECAN. A six-month period of participant observation of various activities hosted by CECAN resulted in the generation of a database of observation notes. The notes covered team dynamics and references to APM principles and practices in four meetings and two workshops, as well as the direct experience of facilitating an agile process for a specific project with four participants from CECAN. The observation notes and direct experience, together with the parallel literature review on agile principles and the adoption of agile practices in a variety of contexts, served as the basis for the development of the structure and areas of analytical focus.

The statistical and content analysis involved group interactions on the digital Kanban tool Trello ( trello.com ). Trello, a web-based project management application, is used as the main channel for coordination and knowledge sharing at CECAN. Data gathered by exporting JSON files and manual scraping of web content from 43 Trello boards facilitated the understanding of patterns of interaction between levels of activity and types of interaction. More specifically, to observe the correlation between the number of active participants, topics covered on each board and relevant actions on cards (change of status, comments and attachments) were analysed. This provided an overview of relevant interactions as well as active projects related to the centre, and allowed for more detailed coverage of the use of digital Kanban boards, which was one of the topics addressed by the interview questions and the data analysis.

An interview protocol, designed as the third and main source of data for the study, was used for seventeen semi-structured interviews with researchers (nine men and eight women) from diverse disciplines and institutions who have experience with the adoption of agile practices in their projects ( Table 1 ). The interview questions were developed with the goal of obtaining different perspectives on the experiences of researchers with the use of agile methods for collaboration in their projects. Using the semi-structured approach, the interviews took the form of conversations guided by questions on APM practices, the scrum framework, teamwork and research activity, which naturally evolved through relevant threads of conversation. The participants varied by field, academic background and experience; some were early-career while others were mid- to late-career. Ten researchers (RC), from several universities and backgrounds who collaborate with CECAN on regular basis, were interviewed. Among them, six interviewees had the specific role of Scrum Master at CECAN, with responsibility for the coordination of various case studies, on which other researchers and stakeholders from various institutions collaborate. The remaining seven interviewees (RE) were researchers and practitioners affiliated with institutions outside CECAN, who also had direct experience in the application of agile principles, to some extent, in research or academic-related projects. These seven additional interviews were conducted in the same period as the other ten, and served as a control group for contrasting diverse observations and for understanding widely important issues derived from interviews to CECAN members.

Researchers and agile practitioners interviewed.

To capture interview data accurately, each interview (which lasted approximately one hour per participant) was audio-recorded and later transcribed for coding. Using a grounded theory approach, data was coded for emerging themes ( Martin and Turner, 1986 ). Themes were discovered through a recursive coding process, then grouped into three areas of inquiry related to the research questions ( Table 2 ): (1) conditions for adopting agile methods in research, (2) adoption of scrum practices and tools, and (3) limitations and advantages of APM adoption in a distributed research organisation. Results were collated into a structured corpus of voices following that sequence, with the most representative and relevant answers selected from interviewees.

Themes derived from interviews in relation to research questions.

The results elaborate the relationship between key principles and practices derived from the literature review on agile methods and principles and reflect the findings based on activity and perceptions of participants, while at the same time integrating a description of the basic features of the scrum framework adapted during its experimental adoption.

In relation to the interviews conducted for this study, although the method as applied in this case does not require an ethical committee approval from the author's institution (Universitat Oberta de Catalunya), all the interviewees volunteered for it and signed a consent for participation accordingly, guaranteeing that confidentiality as a participant will remain secure, and that subsequent uses of records and data will be subject to standard data use policies which protect the anonymity of individuals and institutions.

4.1. Conditions for adopting agile methods in research

From the observations and interviews conducted with CECAN researchers, from the outset, it appears that the underlying rationale for selection was premised on key features of the scrum framework and agile methodology such as flexibility, autonomy and self-organisation.

4.1.1. Complex and changing setting

Since the early operations of CECAN in 2016, its executive board promoted the idea of adopting scrum methods as a possible solution for the self-management of projects, from case studies or workshop organisation to other publication-oriented initiatives. The complexity of conducting research with groups of stakeholders who operate under existing policies, while also setting an evaluation framework for new ones, demonstrates, as one participant observed, that “unpredictable events can come along and change the system potentially” (RC1). This need to regularly adapt activity to a complex context, in a new research institution with more than 30 researchers involved (most of them part-time, and usually collaborating from a range of institutions), also presented a significant management challenge, where it seemed “quite hard for any individual to regularly keep up with all that's going on” (RC2).

4.1.2. Capacity for self-organisation

Another key agile principle relates to the focus on the interactions of self-organised teams. In this case, the scrum framework facilitated regular interaction and feedback among participants. The adoption of the scrum framework was based on the same logic of self-organisation of CECAN, with teams assembled according to the interest or potential contribution of each participant to specific topics, with a logic of combining diverse disciplines and points of view. In this way, as one participant noted, “the vision comes from everyone and it is not like that one person got the direction, it actually emerges from the collective expertise of the group” (RC3).

From the perspective of participants in CECAN case studies, self-motivation was a key factor in many of the parallel projects of the centre, which usually started with very open internal calls:

The initial asking of people who wanted to be involved had to be very open, anyone who thinks they want to help is welcome to. So, I'd have that as a founding principle (RC4).

In this sense, challenges in self-organising, and especially self-assignment in adopting scrum methods for knowledge-based tasks, were cited by other researchers and professionals outside of CECAN.

An ideal scrum team is that which can sit together for a long time and listen to each other. This can significantly augment your learning process. But my theory about research is that you usually don't have this kind of team (RE1).

4.1.3. Flexibility

From observations and interviews with CECAN researchers, the flexibility of the scrum framework seemed to be one of the main reasons why APM principles were considered useful and put into practice: “I felt that this was a way of rationalizing the process that we were already doing and getting it a little bit more structural, while still valuing the flexibility that we had” (RC7).

When interviewed, researchers from outside of CECAN, who have experienced the use of agile practices in academic and research settings, also considered the extent to which it is important to be flexible and start by adapting only some of the scrum principles (to avoid excessive rigidity in its application):

If you take scrum very literal it might not work. For example, if you have divided the project into small areas and manage each one with scrum then it might be very difficult to have four daily meetings in four different groups is an hour of work every day. (RE1).

4.1.4. Adaptivity

Many participants viewed the agile framework as an interesting alternative, and a clear, easy concept to communicate and agree on. It is noteworthy that this occurred in the context of an organisation that deals regularly with the analysis and implementation of methodological approaches in areas of research and evaluation, adapting to different institutional environments and ways of working.

When you use the word ‘agile’, I think people don't question it. I think in a natural language sense, in an English sense, the meaning of the word has relevance and it sounds fine. If you say ‘we're going to work in an agile way’, I think that communicates quickly the idea (RC4).

Researchers interviewed from outside of CECAN also highlighted the importance of “learn by doing” during the initial adaptation of the scrum methodology to their specific domains, realising that it meant a way of approaching management by progressively trying things out:

We were already working with an agile approach but we had not called it ‘agile’. Later on, we started formalizing things and picking up more and more scrum tools and techniques to improve the ways we manage our projects (RE2).

4.2. Adoption of scrum principles and tools

One of the fundamental principles of the self-organising, small operative teams at CECAN was to be innovative at the management level to gain efficiency in collaboration.

We needed to adopt an approach where you can have a consensual decision making that's not necessarily a top down process, but more of a bottom up process of dialogue of mutual interaction (RC6).

It is also important to highlight that CECAN's approach to the adoption of the scrum framework was not based on specific, dedicated training or an expert coach hired for the task. It was instead based more on an evolving interpretation of the APM principles and on experimentation on the basis of an explorative, self-taught approach to the concept.

It was according to what was required and people's individual availability and restraints, and managing that set of interactions. Evidently, we were at each stage constantly thinking about agile. ‘We might have to do this and this. That's what we should do’ (RC6).

4.2.1. Facilitation roles (Scrum masters)

Soon after that initial meeting at which the core principles of the scrum framework were introduced, several of the researchers collaborating with CECAN started to adopt some of its key elements. The role of Scrum Master was one of the principles adopted. At CECAN, the role was conceived as a coordinator for case studies, which had on average four, but up to eight participants ( Fig. 1 ). CECAN Scrum Masters viewed their role as the link between specific tasks and objectives and other collaborator researchers, as well as the liaison with policymakers and representatives from government agencies. This key role was performed by CECAN researchers instead of professional Scrum Masters, and was focused on coordination, facilitating connections and providing guidelines for specific case studies.

Diagram of the scrum adaptation for research and evaluation projects at CECAN.

At CECAN, Scrum Masters were seen as connectors of expertise and coordinators guided by shared goals, as one participant explained, “I see the role of Scrum Master as a kind of intermediary in an interdisciplinary project like this one. The expertise that a Scrum Master acquires is in linking an expert to an expert and that itself requires [a] particular set of expertise for CECAN, this is an ongoing challenge” (RC9).

In this sense, the role of Scrum Master could be considered an appropriation and reinterpretation. It was one of the key agile practices adopted at CECAN, and although perceived by some participants as not fully implemented, the Scrum Master seemed to play a critical facilitation role and contributed to expanding agile management practices to the various research initiatives and projects. As corroborated by the following comment from one participant, the facilitation role provided participants with transparency and guidance, as opposed to a command and control approach, as they engaged in joint activities.

The role is very much one of a leading rather than controlling. That has to be the case because there's actually quite a lot of skill involved in managing a group of researchers for whom you have … to align management responsibility. We are a consortium of fourteen different academic organisations. If I wanted to tell you or anybody else in the team ‘you have to do this, … because I'm telling you to’ they will just go away (RC2).

Considering the high volume of case studies, publications and other tasks related to CECAN activity, for researchers acting as Scrum Masters there was also the opportunity to learn from colleagues doing the same, or even to share the role:

[In a specific project] There's really two of us acting as Scrum Master because we're covering a broad complex area of policy, to which both of us bring complementary experience. So, he and me communicate, I would say, daily. With other colleagues in CECAN, usually it's once a week at least (RC1).

It is also significant the extent to which the responsibility of having a facilitation and coordination role required additional networking efforts and expertise from researchers new to the concept:

[The Scrum Master role] It was slow to develop initially. I think it was partly about building trust and establishing relationships with the policy partners, and deciding what they wanted out of the process, and really getting a grip of what they wanted to do, how they wanted to work with CECAN (RC10).

4.2.2. Kanban boards

At CECAN—a “distributed virtual organization, with so many people doing so many things with different time involvement” (RC5)—the Kanban board was one of the agile management practices adopted. The CECAN boards were digital and created using Trello, a web-based project management application, in a format replicating Post-It notes ( Fig. 2 ). The Trello boards were one of the main channels of documentation for the centre. They were managed mainly by the Scrum Masters, and were accessed by the other CECAN researchers and occasionally by external collaborators or other stakeholders.

Screenshot of one of the CECAN Trello boards, with different tasks on cards.

As explained in the excerpt below, each new initiative or discussion was eventually translated into modular pieces of information. This represents a novel way of accessing updated and valuable knowledge for the entire organisation about the progress of projects.

One of the ways that CECAN is trying to adopt an agile approach was to set up the use of Trello boards and the use of Trello as a system for those who were engaging in case studies, but also those who were engaging in non-case study activities. To update not just their own group, but the rest of CECAN as well. The use of Trello was a way of leading the case studies, updating data for example with case study notes, and what was happening on the case studies, and any particular event that was going on (RC6).

The results from a basic statistical analysis of communication and interactions on the various Trello boards at CECAN ( Fig. 3 ) suggest that there is a relative correlation between more active users on each board, the number of cards assigned to participants and activity related to assigned cards (usually displacing them on the board according to workflows, or content edits). There appeared to be no correlation with publishing comments on cards and attaching documents to cards, as this occurred less regularly, apart from some exceptions. This would confirm that the Trello tool was used consistently through the different boards and related projects, following the typical APM process for visualizing workflows. On average, however, the analysis of the aggregated data shows that only a minority of researchers were active on the Trello platform (despite the entire organisation having full access to all the boards), which represents an unequal distribution of participation.

Statistic of cards on each Trello board, ordered by number of active users.

A comparison of the most active Trello boards, an analysis of the different levels of engagement with the tool, as well as an observation of the progressive familiarization with its functionalities and connection to APM principles, revealed that participants generally viewed their experience in using the Trello boards as an evolving process parallel to the levels of intensity and activity in the organisation. This observation is supported by the following statement from a participant who was less active in interacting or generating cards, but benefited from accessing the Trello boards: “It has proved a useful kind of map of how the case studies have evolved, sort of a narrative, if you like, a narrative of kind of key points within each of the case studies and how they developed,” (RC10).

Trello was generally perceived as practical and aligned with the need to specify, visualize and assign tasks for case studies or publications, and also “useful to have a quick overview of what is happening, and to understand what other people are doing in a quick way” (RC4). However, it represented a way of working and adapting to a specific type of interface with a significant learning curve, to which not all researchers found it easy to adapt:

I have done a lot of different projects within different project management and communication tools and it becomes too complicated in my mind (RC9).

This coincides with experiences from other researchers, and the following excerpt highlights that some colleagues, perhaps on the basis of their digital literacy, perceive this type of tool as a barrier:

There was a senior researcher struggling through it and ended up in chaos. She did not want to adapt to these things. If I said ‘Put this on the Trello board, we do not need to implement the whole thing, we can manage with something’ her response would be like, ‘Oh! What is this? I do not want to install this. I do not want to join this, it's complicated’ (RE1).

4.2.3. Development sprints

With respect to the adoption of scrum methods at CECAN, the concept of sprints was less explicit or used among participants. For example, the practice of establishing regular “standup” meetings, or retrospective meetings at the end of each sprint period was not routinely followed. Instead, researchers usually established collective agreements about the duration and responsibilities related to specific tasks, depending on the project.

Like other key aspects of agile management practices, this sprint principle —although not used with the same rigour as in software development contexts— was progressively incorporated into the logic of shared communication at CECAN:

All of those things require structured communication baseline and tasks and milestone baseline. The point is not only moving forward but also ensuring that we are still understanding each other with constant feedback (RC3).

However, from some comments emanating from the interviews, the sprint also seems to be a problematic concept to appropriate from software development processes and to adopt for the peculiarity of research tasks:

Usually two weeks long, I think everyone would agree that this is how long a sprint should be. I found it funny because probably in the tech world it works, but when you have a different type of tasks the two-week period is a bit arbitrary. In one of our case studies I had workshops which were organized about one month in advance, which made a very good sense of working in sprints (RC7).

When compared with other practitioners and researchers with longer experience using agile methods in non-software contexts, there seems to be a significant difference in the way sprints were adopted at CECAN and how they were experienced in other cases, where they constituted a central part of the process:

You just don't do one sprint, it is more like doing sprints after sprints after sprints. By doing that and looking at things in many different ways, we get incredible depth (RE3).

Sometimes we block entire evenings without any other task, or plan one-day trips to finish an article with another author. Then 10 hours working and although the article is not over, it is properly drafted (RE4).

4.2.4. Incremental development

It should also be noted that the concept of incremental development by small and regular releases (derived from the Lean principle of “minimum viable products”), when initially adopted from software development, was perceived as another complex approach to be tackled in the context of academic research.

We do work considering minimum viable products in a way, by thinking about stages of our work. For example, from a case study to a paper, and all the steps in between. But we don't use these exact words, is more implicit than explicit (RC5).

However, this practice, once adopted, served as an inspiration or key principle for some participants. As with the principle of sprints, some researchers developed an understanding and progressive process of adaptation of the concept of incremental research results, particularly in relation to the other organisations and stakeholders with whom they collaborated. As two participants observed:

What became quite useful I think in the use of the agile approach with [Stakeholder organisation A] was to say ‘we're going to iterate. We know right from the beginning there's going to be a lot of iterations'. To be able to describe that to them at the beginning. They never expected a final product suddenly to appear out of nowhere (RC4).

We start out with a set of objectives, but we have to adapt along the way. The set of objectives might change or how we meet those objectives might need to change, following the idea of continuous and feedback loops. That's how we're working in collaboration with [Stakeholder organisation B], but we're also having to touch base with them on a regular basis, because things are changing (RC1).

Outside of CECAN, other researchers also expressed familiarity with the principle, with some researchers even adopting and adapting the concept for use in their own research findings and academic writing.

‘Minimal viable finding’ is related to the way we are able to focus our research in every two weeks timeframes. ‘Here are things which are more promising and we are going to focus in this process’. We usually find several things but this is about highlighting something which we are going to promise and are going to deliver (RE3).

4.3. Challenges of APM adoption in a distributed research organisation

4.3.1. need for balance.

Given the special nature of research activity, and the need for flexibility in terms of allowing experts to add value through their knowledge and expertise, there was a constant effort at CECAN to balance, adaptively, the need to produce results and to advance in the evaluation of policies without “having a hierarchical [structure of] control” (RC2). In this sense, some participants perceived the achievement of this balance as one of the most challenging aspects of assimilating new coordination approaches, and the need of leadership through the Scrum framework as a key factor for providing results without sacrificing autonomy. For some participants, when compared with the expectations implied by APM self-organisation, these attempts appeared to be not always successful.

4.3.2. Offline vs online context

It is also important to highlight at this stage that there are few opportunities, in the context in which CECAN operates as a distributed research initiative, to meet offline in face-to-face meetings, with the result that facets such as self-organisation become more complicated (and usually require varying levels of online interaction), as described by this respondent:

I feel the biggest challenges with me while trying to do agile and scrum with CECAN is that we are remote, so it is difficult to have the immediate emergency or urgency of something that I need to do, compared to if you are seeing someone in person (RC7).

Limitations in team size and difficulties in adapting online because of individual research styles were also viewed as key issues that need to be resolved for agile methods to function effectively in this context.

[A specific publication project] started with probably 12 people who were interested but it was very difficult to get momentum of any kind. Everyone was interested in being involved but there was no momentum to start doing anything. So, in discussion with A., we decided to make the group much smaller to just three members. After this change, we have been working smoothly (RC7).

4.3.3. Proliferation of kanban boards

Due to the initial recommendations on the use of Trello at CECAN, the boards were used for the management of various types of projects, and not only case studies for publication but also for planning of workshops, the design of new methodologies or the evaluation of policies. As a result, there seemed to be a proliferation of boards, which were not always useful or used in accordance with agile principles. One participant expressed the following view:

I think that Trello works best where people have defined responsibilities for a board and know who to contact, plus have predefined rules which are particular to a board. Probably, this concept has not been as clear to the users as it could have been, partially due to the fact that it is a new concept for everyone (RC7).

The experiences recounted by participants on the use of Trello boards as a discussion channel reflect their expectations about the tool in relation to their communication needs, given the complexity and limits of interchanging knowledge from their individual locations and institutions. Others highlighted the difficulty of adopting new digital tools instead of developing new strategies focused on the physical context.

4.3.4. Trust in relationships

The high volume of case studies, publications and other tasks reflected on the numerous Trello boards, afforded researchers acting as Scrum Masters the opportunity to learn from colleagues in similar positions, or for sharing the role, and thereby learn about the implications of managing case studies as Scrum Masters in a more networked and interactive way. However, as far as the responsibility of the facilitation and coordination role is concerned, as one participant explained, the extent to which it required additional effort and progressive ‘learn-by-doing’ expertise from researchers new to the concept is significant.

It was slow to develop initially. I think it was partly about building trust and establishing relationships with the policy partners, and deciding what they wanted out of the process, and really getting a grip of what they wanted to do, how they wanted to work with CECAN (RC10)

This view is similar to that of other researchers who experienced the same challenges in similar roles in research-oriented or academic contexts other than CECAN.

4.3.5. Types of research

In some of the interviews, there was often a return to the question about the extent to which it is possible to adopt agile principles in all types of research or whether, as in the view of some CECAN participants, APM principles represent a methodological framework that is more suitable to applied research and contexts where time constraints and pressure from stakeholders make it more applicable and imperative.

There are projects which are quite theoretical, with basic research, where this kind of agile is probably not likely to be very helpful. So, I wouldn't want to force agile on every piece of research (RC2).

In contrast, confirming interest in the scrum framework from a wider perspective, in front of the same question other interviewees commented how APM practices were incorporated into their own research organisations, outside of their collaboration with CECAN:

I have seen it working nicely across a variety of domains. In my small department we started it from zero, we have been doing it and have witnessed it progress. Now, we are about ten people (RC3).

I am working with two other people on projects who are not part of the original CECAN team, they have been subcontracted to come in and help work on it. I have been ‘scrumming’ with them offline, not using the traditional forums like Trello (RC7).

4.3.6. Ad hoc adoption

However, other perspectives also addressed the complexity of applying agile principles to CECAN research and evaluation outputs and the key limitations of time and resources, as well as its correlation with the need for more flexibility in coordination:

I think adopting an agile approach in a prescriptive way it's not necessarily effective. It's easy to be quite agile in the sense of having a very weekly sense of meetings, at a particular time on a particular day, if the people that are involved are not overly constrained in terms of time or labour or any sort of resource constraints. When they are, then you have to be quite adaptable or flexible according to the regularity, according to the main principal parties involved. So, I think from that perspective, more open agile use approach is perhaps more effective than a prescript one that says ‘we're going to be this regular in terms of when our meetings are going to happen, when we need to update the Trello board, so on and so forth’ (RC6).

In this respect, other experts with experience in the utilisation of agile principles outside of CECAN also emphasized the importance of flexibility and openness when adopting these methodologies, instead of following blindly the rules and proceedings as they are established in software development processes.

I have come up with methodologies, and I know that they're all made up, there are frictional of context specific tricks, and methods, and tools and thinking. Agile presupposes that the ‘big box’ methodologies can ignore context in a way. Like a call and response mechanism which is very rule based and explicit, and I don't think that this is how a method works (RE5).

4.3.7. Institutional culture

For other researchers, who are familiar with scrum and agile methods, another key issue is related to the complexity and the management challenges embedded at the institutional level in universities and scientific departments:

The group can be agile but it faces a system like the academia and the university that is not agile. So, the motivation to do research and at the same time adapt to new ways of doing is complicated to manage (RE4).

Managers of research projects and IPs are not trained in project management, nor these skills are covered in PhD courses or similar. You can only self-learn about it, or explore on your own your ability to do so by acquiring collaboration skills and techniques (RE7).

5. Discussion

The objective of this article was to explore the adoption of agile methods in a distributed research initiative, and especially the appropriation of the scrum framework as a coordination and communication solution for the management of collaborative interdisciplinary projects. Taking into account the specific characteristics applicable to research in academic and scientific areas (as a separate context from software development processes, where the APM framework was developed and is widely used), the adoption seemed successful overall in that it facilitated the generation of new dynamics of collaboration, benefiting from some APM principles and practices in various ways. However, the process was also challenging and had some limitations in terms of a shared understanding and coherent application of the scrum framework, when compared to similar experiences in the use of agile methods in research projects.

In this regard, according to the data obtained from interviews, the adoption of agile methods in research collaboration is suited to organisations embedded in complex and changing settings, with some capacity for self-organisation, flexibility and adaptivity to new management approaches, which connects with the description of organizational networks ( Spinuzzi, 2015 , p. 58). On the other hand, relevant challenges identified for APM adoption in research point to issues related to: (1) the needed balance between efficiency and autonomy of participants, (2) the limitations of the online context for coordinating activity, (3) the tendency to proliferation of kanban boards; (4) the need to build trust in relationships when coordinating, (5) the type of research activity carried out, (6) time and resources constraints, (7) the importance of tailoring scrum principles to activities, and (8) the institutional culture of academic and research organisations.

Integrating agile methods and practices for interdisciplinary collaboration requires high degrees of flexibility and “learn by doing” approaches, similar to other project management methodologies and approaches ( Lauren, 2018 , p. 30). In this sense, the scrum framework constitutes a methodological framework that can be counterproductive if it is too ambitiously or rigidly implemented in this type of context, as indicated in the literature on the utilisation APM outside of the software development sector ( Ciric et al., 2018 ). According to Nonaka and Takeuchi (1995) , this type of participative management can be favourable for several types of agile development where conditions such as “built-in instability, self-organizing project teams, overlapping development phases, ‘multilearning’, subtle control, and organizational transfer of learning”, converge and are present to some extent in the philosophy of the collaboration initiative. When adopted by academic participants and experts familiar with research or evaluation methods, the scrum framework seems to be an easy concept to transfer and experiment with, even though specific tailoring to the idiosyncrasies of collaboration and personal motivations may be required when adapting APM ( Gandomani et al., 2014 ). Also, as attested by the literature on agile software development, characteristics such as team size and specificities such as the online tools required for operating in distributed contexts seem critical, as well as its suitability for small groups but not for large projects ( Cohen et al., 2004 ), or the significant complexity that may be experienced when adopted by remote as opposed to collocated project teams ( Paasivaara et al., 2009 ; Teasley et al., 2000 ).

Scrum principles adopted by various research teams, as analysed in this study, were seen as a valuable addition to the coordination of projects, with diverse levels of agreement about their successful implementation and perceived challenges. For CECAN self-organised teams, in a networked context requiring new participation strategies, working on case studies following APM principles provided a structured approach to a different style of management of evaluation and research-related tasks. Teams perceived positive attributes that are also referenced in previous studies about agile methods, including easy adoption and relation to project success ( Serrador and Pinto, 2015 ), as well as improved teamwork through the focus on human and social factors ( Dybå and Dingsøyr, 2008 ). Several interviewees highlighted the key role of the Scrum Master as facilitator but showed less agreement in relation to new concepts when applied to scientific activity such as “sprint development”, or the importance of small and regular releases of research outputs, when applied to scientific activity.

Studies on agile management have demonstrated the benefits to be gained with respect to fostering trust and cohesion in teams ( McHugh et al., 2012 ). Empirical evidence points to a correlation with differing levels of shared leadership, team orientation, cross-functionality, internal learning processes and team autonomy ( Moe et al., 2009 ; Stettina and Heijstek, 2011 ). This seems to be the case as well in the specific research context studied at CECAN, and also when contrasted with perspectives from other researchers who are familiar with agile methods. Some of the limitations of agile methods addressed by academic literature are also present in this case, such as the difficulties experienced by certain individuals or personality types in properly integrating into agile teams ( Whitworth and Biddle, 2007 ). As well as the constraints perceived as inherent to the tradition of academic institutions and the lack of new management practices in scientific activity ( Pope-Ruark, 2017 ), or difficulties in adapting to digital tools by senior researchers, some other complexities of adopting agile methods for research were evident. For instance, the timeframes for developing intellectual activity, and the motivation for doing so, can vary significantly depending on the type of project. Also, some researchers held the view that there should be a balance between prescriptive and adaptable formulas for this type of dynamic management.

In relation to specific tools, only a relative minority of researchers were active on the Trello platform, despite the entire organisation having full access to all the boards. This unequal distribution of participation via the digital Kanban board seems to represent a typical “90/9/1 principle” or “power law” ( Nielsen, 2006 ), usually present in online communities of peer production, where the fact that a large percentage of people do not contribute does not necessarily constitute a problem or put at risk the achievement of common goals ( Fuster Morell, 2014 ). In this sense, for a number of researchers, the proliferation of Trello boards represented an organisational challenge in terms of managing the tasks in progress and staying on top of all the boards, once several boards were in active use, which coincides with the findings of other studies about the adoption of digital Kanban boards for knowledge management in distributed organisations ( McLean and Canham, 2018 ).

Lessons learned from this case study point to the need to reconsider the suitability of the scrum framework as the best agile approach for distributed research management. Future studies should explore if more open interpretations of APM practices (which for example focus on the regular but less structured updating of tasks via Kanban boards) could be more successfully adopted in this context, or if on the contrary, additional scrum practices (such as regular “standups” in short periods, or retrospective meetings) could improve the adaptation of APM principles and practices adapted to research activity. Another relevant issue emanating from this exploratory study relates to whether the adoption of professional agile facilitation (by experts in scrum or other agile practices and not researchers) is important and should be addressed with a comparative focus in future cases. As one of its main limitations, this study did not gather data that could compare adoption in such terms. Finally, in relation to the critical factor of remote, distributed research teamwork, another line of inquiry should address how agile practices could be used effectively in fully allocated science teams, where sharing the same physical space could benefit from the use of offline Kanban boards, as opposed to digital ones.

Declarations

Author contribution statement.

Enric Senabre Hidalgo: Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This work was supported by CECAN and University of Surrey under a Fellowship grant agreement.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.

• Abrahamsson P., Salo O., Ronkainen J., Warsta J. 2017. Agile Software Development Methods: Review and Analysis. arXiv preprint arXiv:1709.08439. [ Google Scholar ]
• Ahmad M.O., Markkula J., Oivo M. Software Engineering and Advanced Applications (SEAA), 2013 39th EUROMICRO Conference on. IEEE; 2013. Kanban in software development: a systematic literature review; pp. 9–16. [ Google Scholar ]
• Anderson D.J., Concas G., Lunesu M.I., Marchesi M., Zhang H. International Conference on Agile Software Development. Springer; Berlin: 2012. A comparative study of Scrum and Kanban approaches on a real case study using simulation; pp. 123–137. [ Google Scholar ]
• Armstrong P., Gordon R., Hoffecker E., Krystalli R., Leith K., Stinchfield B., Wilson K. MIT; Cambridge, MA: 2015. Lean Research: Defining Rigor. Working Paper, MIT D-Lab-The Fletcher School of Law and Diplomacy. [ Google Scholar ]
• Bammer G. Enhancing research collaborations: three key management challenges. Res. Pol. 2008; 37 (5):875–887. [ Google Scholar ]
• Barroca L., Sharp H., Salah D., Taylor K., Gregory P. Bridging the gap between research and agile practice: an evolutionary model. Int. J. Syst. Assur. Eng. Manag. 2015:1–12. [ Google Scholar ]
• Beck K., Beedle M., van Bennekum A., Cockburn A., Cunningham W., Fowler M., Kern J. 2001. Manifesto for Agile Software Development. http://agilemanifesto.org/ Retrieved from. [ Google Scholar ]
• Borgman C.L. MIT Press; 2010. Scholarship in the Digital Age: Information, Infrastructure, and the Internet. [ Google Scholar ]
• Bozeman B., Corley E. Scientists' collaboration strategies: implications for scientific and technical human capital. Res. Pol. 2004; 33 (4):599–616. [ Google Scholar ]
• Bridgeford T., Amant K.S. Taylor & Francis; 2017. Academy-Industry Relationships and Partnerships. [ Google Scholar ]
• Cao L., Mohan K., Xu P., Ramesh B. A framework for adapting agile development methodologies. Eur. J. Inf. Syst. 2009; 18 (4):332–343. [ Google Scholar ]
• Cervone H.F. Understanding agile project management methods using scrum. OCLC Syst. Serv. Int. Dig. Libr. Persp. 2011; 27 (1):18–22. [ Google Scholar ]
• Ciric D., Lalic B., Gracanin D., Palcic I., Zivlak N. 2018 IEEE International Symposium on Innovation and Entrepreneurship (TEMS-ISIE) IEEE; 2018, March. Agile project management in new product development and innovation processes: challenges and benefits beyond software domain; pp. 1–9. [ Google Scholar ]
• Cocco L., Mannaro K., Concas G., Marchesi M. International Conference on Agile Software Development. Springer; Berlin: 2011, May. Simulating kanban and scrum vs. waterfall with system dynamics; pp. 117–131. [ Google Scholar ]
• Cockburn A. Agile software development joins the “Would-Be” crowd. Cut. IT J. 2002; 15 (1):6–12. [ Google Scholar ]
• Cohen D., Lindvall M., Costa P. An introduction to agile methods. Adv. Comput. 2004; 62 :1–66. [ Google Scholar ]
• Conboy K., Fitzgerald B. Proceedings of the 2004 ACM Workshop on Interdisciplinary Software Engineering Research. ACM; 2004, November. Toward a conceptual framework of agile methods: a study of agility in different disciplines; pp. 37–44. [ Google Scholar ]
• Conforto E.C., Salum F., Amaral D.C., da Silva S.L., de Almeida L.F.M. Can agile project management be adopted by industries other than software development? Proj. Manag. J. 2014; 45 (3):21–34. [ Google Scholar ]
• Cooke N.J., Hilton M.L., editors. Enhancing the Effectiveness of Team Science. National Academies Press; Washington, D.C.: 2015. [ PubMed ] [ Google Scholar ]
• Daradoumis T., Roca M., Grasman S.E., Faulin J. Collaborative and distributed e-research: innovations in technologies, strategies, and applications. In: Juan A.A., editor. Information Science Reference. 2012. [ Google Scholar ]
• Delfanti A. Users and peers. From citizen science to P2P science. Cell. 2016; 21 :01. [ Google Scholar ]
• Demir S.T., Theis P. 24th Annual Conference of the International Group for Lean Construction, Boston, USA. 2016. Agile design management–the application of scrum in the design phase of construction projects. [ Google Scholar ]
• DeWit T. Lean techniques: the QC lab can reduce product lead times. Qual. Assur. J. 2011; 14 (3-4):72–75. [ Google Scholar ]
• Dybå T., Dingsøyr T. Empirical studies of agile software development: a systematic review. Inf. Softw. Technol. 2008; 50 (9):833–859. [ Google Scholar ]
• Eccles K., Schroeder R., Meyer E.T., Kertcher Z., Barjak F., Huesing T., Robinson S. Proceedings of NCeSS International Conference on e-Social Science, Cologne, June. 2009. The future of e-research infrastructures; pp. 24–26. [ Google Scholar ]
• Fuster Morell M. Governance of online creation communities for the building of digital commons: viewed through the framework of the institutional analysis and development. In: Madison M.J., Strandburg K., Frischmann B., editors. Governing the Knowledge Commons. Oxford University Press; Oxford: 2014. [ Google Scholar ]
• Fernandez D.J., Fernandez J.D. Agile project management agilism versus traditional approaches. J. Comput. Inf. Syst. 2008; 49 (2):10–17. [ Google Scholar ]
• Friess E. “Filling to capacity”: an exploratory study of project management language in agile scrum teams. Tech. Commun. 2018; 65 (2):169–180. [ Google Scholar ]
• Gandomani T.J., Zulzalil H., Ghani A.A., Sultan A.B.M., Sharif K.Y. How human aspects impress Agile software development transition and adoption. Int. J. Softw. Eng. Appl. 2014; 8 (1):129–148. [ Google Scholar ]
• Gray D.O., Lindblad M., Rudolph J. Industry–university research centers: a multivariate analysis of member retention. J. Technol. Transf. 2001; 26 (3):247–254. [ Google Scholar ]
• Grimheden M.E. Can agile methods enhance mechatronics design education? Mechatronics. 2013; 23 (8):967–973. [ Google Scholar ]
• Hannola L., Friman J., Niemimuukko J. Application of agile methods in the innovation process. Int. J. Bus. Innov. Res. 2013; 7 (1):84–98. [ Google Scholar ]
• Hekler E.B., Klasnja P., Riley W.T., Buman M.P., Huberty J., Rivera D.E., Martin C.A. Agile science: creating useful products for behavior change in the real world. Transl. Behav. Med. 2016; 6 (2):317–328. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Helbing D., Brockmann D., Chadefaux T., Donnay K., Blanke U., Woolley-Meza O., Perc M. Saving human lives: what complexity science and information systems can contribute. J. Stat. Phys. 2015; 158 (3):735–781. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Hicks M., Foster J.S. Department of Computer Science; 2010. Adapting Scrum to Managing a Research Group. http://www.cs.umd.edu/∼mwh/papers/score.pdf Technical Report #CS-TR-4966. Retrieved from. [ Google Scholar ]
• Hoda R., Noble J., Marshall S. Self-organizing roles on agile software development teams. IEEE Trans. Softw. Eng. 2013; 39 (3):422–444. [ Google Scholar ]
• Jeffrey P. Smoothing the waters: observations on the process of cross-disciplinary research collaboration. Soc. Stud. Sci. 2003; 33 (4):539–562. [ Google Scholar ]
• Jirotka M., Lee C.P., Olson G.M. Supporting scientific collaboration: methods, tools and concepts. Comput. Support. Coop. Work. 2013; 22 (4-6):667–715. [ Google Scholar ]
• Katz J.S., Martin B.R. What is research collaboration? Res. Pol. 1997; 26 (1):1–18. [ Google Scholar ]
• Keijzer-Broers W.J., de Reuver M. vol. 9661. Springer; NY: 2016. Applying Agile design sprint methods in action design research: prototyping a health and wellbeing platform; pp. 68–80. (Proceedings of the 11th International Conference on Tackling Society's Grand Challenges with Design Science). [ Google Scholar ]
• Keraminiyage K., Haigh R., Amaratunga D. Achieving success in collaborative research: the role of virtual research environments. J. Inf. Technol. Constr. 2009; 14 :59–69. [ Google Scholar ]
• König B., Diehl K., Tscherning K., Helming K. A framework for structuring interdisciplinary research management. Res. Pol. 2013; 42 (1):261–272. [ Google Scholar ]
• Kraut R.E., Galegher J., Egido C. Relationships and tasks in scientific research collaboration. Hum. Comput. Interact. 1987; 3 (1):31–58. [ Google Scholar ]
• Küpper S. Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering. ACM; NY: 2016. The impact of agile methods on the development of an agile culture: research proposal: [the agile evolution] p. 1. [ Google Scholar ]
• Lauren B. 2018. Communicating Project Management: a Participatory Rhetoric for Development Teams. Routledge. [ Google Scholar ]
• Lassi M., Sonnenwald D.H. vol. 15. 2010. Identifying factors that may impact the adoption and use of a social science collaboratory: a synthesis of previous research. (The Seventh International Conference on Conceptions of Library and Information Science (CoLIS)—“Unity in Diversity”). No. 3. [ Google Scholar ]
• Lehnen J., Schmidt T.S., Herstatt C. Bringing agile project management into lead user projects. Int. J. Prod. Dev. 2016; 21 (2-3):212–232. [ Google Scholar ]
• Lei H., Ganjeizadeh F., Jayachandran P.K., Ozcan P. A statistical analysis of the effects of scrum and kanban on software development projects. Robot. Comput. Integr. Manuf. 2017; 43 :59–67. [ Google Scholar ]
• Liao C.H. How to improve research quality? Examining the impacts of collaboration intensity and member diversity in collaboration networks. Scientometrics. 2011; 86 (3):747–761. [ Google Scholar ]
• Lima I.R., de Castro Freire T., Costa H.A.X. Adapting and using scrum in a software research and development laboratory. Rev. Sist. Inf. FSMA. 2012;(9):16–23. [ Google Scholar ]
• Mara A.F., Potts L., Bartocci G. Proceedings of the 31st ACM International Conference on Design of Communication. ACM; NY: 2013. The ethics of agile ethnography; pp. 101–106. [ Google Scholar ]
• Marcal A.S.C., Soares F.S.F., Belchior A.D. Software Engineering Workshop, 2007. SEW 2007. 31st IEEE. IEEE; Washington, D.C: 2007, March. Mapping CMMI project management process areas to SCRUM practices; pp. 13–22. [ Google Scholar ]
• Marchesi M., Mannaro K., Uras S., Locci M. Distributed scrum in research project management. In: Concas G., Damiani E., Scotto M., Succi G., editors. Agile Processes in Software Engineering and Extreme Programming. XP 2007. vol. 4536. Springer; Berlin: 2007. pp. 240–244. (Lecture Notes in Computer Science). [ Google Scholar ]
• Martin P.Y., Turner B.A. Grounded theory and organizational research. J. Appl. Behav. Sci. 1986; 22 (2):141–157. [ Google Scholar ]
• McHugh O., Conboy K., Lang M. Agile practices: the impact on trust in software project teams. IEEE Softw. 2012; 29 (3):71–76. [ Google Scholar ]
• McLean J., Canham R. Managing the electronic resources lifecycle with kanban. Open Inf. Sci. 2018; 2 (1):34–43. [ Google Scholar ]
• Moe N.B., Dingsøyr T., Røyrvik E.A. Putting agile teamwork to the test –a preliminary instrument for empirically assessing and improving agile software development. In: Abrahamsson P., Marchesi M., Maurer F., editors. Agile Processes in Software Engineering and Extreme Programming. XP 2009. vol. 31. Springer; Berlin: 2009. pp. 114–123. (Lecture Notes in Business Information Processing). [ Google Scholar ]
• Münch J., Fagerholm F., Johnson P., Pirttilahti J., Torkkel J., Jäarvinen J. Lean Enterprise Software and Systems. vol. 167. Springer; Berlin: 2013. Creating minimum viable products in industry-academia collaborations; pp. 137–151. (Lecture Notes in Business Information Processing). [ Google Scholar ]
• Nielsen J. Participation inequality: lurkers vs. contributors in internet communities. Jakob Nielsens Alertbox. 2006 [ Google Scholar ]
• Niemi-Grundström M. Developing, evaluating and managing library with agile methods. Libr. Manag. 2014; 35 (6/7):481–485. [ Google Scholar ]
• Nonaka I., Takeuchi H. Oxford University Press; NY: 1995. The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation. [ Google Scholar ]
• Ota M. Scrum in research. In: Luo Y., editor. Cooperative Design, Visualization and Engineering Proceedings of the 7th International Conference (CDE 2010) Springer; Berlin: 2010. pp. 109–116. [ Google Scholar ]
• Paasivaara M., Durasiewicz S., Lassenius C. 4th International Conference on Global Software Engineering, ICGSE'09. IEEE; Washington, D.C.: 2009. Using scrum in distributed agile development: a multiple case study; pp. 195–204. [ Google Scholar ]
• Perkmann M., Walsh K. The two faces of collaboration: impacts of university-industry relations on public research. Ind. Corp. Change. 2009; 18 (6):1033–1065. [ Google Scholar ]
• Pope-Ruark R. University of Chicago Press; 2017. Agile Faculty: Practical Strategies for Managing Research, Service, and Teaching. [ Google Scholar ]
• Powell W.W., Snellman K. The knowledge economy. Ann. Rev. Sociol. 2004:199–220. [ Google Scholar ]
• Rigby D.K., Sutherland J., Takeuchi H. Harvard Business Review; 2016. Embracing Agile. May. [ Google Scholar ]
• Rigby J., Edler J. Peering inside research networks: some observations on the effect of the intensity of collaboration on the variability of research quality. Res. Pol. 2005; 34 (6):784–794. [ Google Scholar ]
• Robinson H., Sharp H. The social side of technical practices. In: Baumeister H., Marchesi M., Holcombe M., editors. Extreme Programming and Agile Processes in Software Engineering. XP 2005. vol. 3556. Springer; Berlin: 2005. pp. 100–108. (Lecture Notes in Computer Science). [ Google Scholar ]
• Rosenberg D., Stephens M. Apress; NY: 2003. Extreme Programming Refactored: the Case against XP. [ Google Scholar ]
• Sandberg A.B., Crnkovic I. Proceedings of the 39th International Conference on Software Engineering: Software Engineering in Practice Track. IEEE Press; Washington, D.C: 2017. Meeting industry: academia research collaboration challenges with agile methodologies; pp. 73–82. [ Google Scholar ]
• Santos N., Fernandes J.M., Carvalho M.S., Silva P.V., Fernandes F.A., Rebelo M.P., Machado R.J. International Conference on Computational Science and its Applications. Springer International Publishing; NY: 2016. Using scrum together with UML models: a collaborative university-industry R&D software project; pp. 480–495. [ Google Scholar ]
• Scrum Alliance . 2016. The State of Scrum Report 2017 Edition. https://www.scrumalliance.org/learn-about-scrum/state-of-scrum/2017-state-of-scrum Retrieved from. [ Google Scholar ]
• Senabre Hidalgo E. Management of a multidisciplinary research project: a case study on adopting agile methods. J. Res. Pract. 2018; 14 (1):2. [ Google Scholar ]
• Serrador P., Pinto J.K. Does agile work? – a quantitative analysis of agile project success. Int. J. Proj. Manag. 2015; 33 (5):1040–1051. [ Google Scholar ]
• Sharp H., Robinson H., Petre M. The role of physical artefacts in Agile software development: two complementary perspectives. Interact. Comp. 2009; 21 (1–2):108–116. [ Google Scholar ]
• Smykla E., Zippel K. OISE; 2010. Literature Review: Gender and International Research Collaboration; p. 936970. Report prepared with funding from NSF. [ Google Scholar ]
• Sonnenwald D.H. Scientific collaboration. Ann. Rev. Inf. Sci. Technol. 2007; 41 (1):643–681. [ Google Scholar ]
• Spinuzzi C. University of Chicago Press; 2015. All Edge: Inside the New Workplace Networks. [ Google Scholar ]
• Stare A. Agile project management in product development projects. Proc. Soc. Behav. Sci. 2014; 119 :295–304. [ Google Scholar ]
• Stettina C.J., Heijstek W. European Conference on Software Process Improvement. Springer; Berlin, Heidelberg: 2011, June. Five agile factors: helping self-management to self-reflect; pp. 84–96. [ Google Scholar ]
• Sutherland J., Altman I. Agile Conference, 2009. AGILE'09. (pp. 350-355) IEEE; 2009, August. Take no prisoners: how a venture capital group does scrum. [ Google Scholar ]
• Teasley S., Covi L., Krishnan M.S., Olson J.S. Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work. ACM; 2000, December. How does radical collocation help a team succeed? pp. 339–346. [ Google Scholar ]
• Veneziano V., Rainer A.W., Haider S. 2014. When Agile is Not Good Enough: an Initial Attempt at Understanding How to Make the Right Decision. arXiv preprint arXiv:1402.5557. [ Google Scholar ]
• Vom Brocke J., Lippe S. Managing collaborative research projects: a synthesis of project management literature and directives for future research. Int. J. Proj. Manag. 2015; 33 (5):1022–1039. [ Google Scholar ]
• Wagner C.S., Whetsell T.A., Leydesdorff L. Growth of international collaboration in science: revisiting six specialties. Scientometrics. 2017; 110 (3):1633–1652. [ Google Scholar ]
• Wang J., Hicks D. Scientific teams: self-assembly, fluidness, and interdependence. J. Inf. 2015; 9 (1):197–207. [ Google Scholar ]
• West D., Grant T., Gerush M., D'silva D. Agile development: mainstream adoption has changed agility. Forrester Res. 2010; 2 (1):41. [ Google Scholar ]
• Whitworth E., Biddle R. Agile Conference (AGILE) IEEE; Washington, D.C.: 2007. The social nature of agile teams; pp. 26–36. [ Google Scholar ]
• Wuchty S., Jones B.F., Uzzi B. The increasing dominance of teams in production of knowledge. Science. 2007; 316 (5827):1036–1039. [ PubMed ] [ Google Scholar ]
• Ynalvez M.A., Shrum W.M. Professional networks, scientific collaboration, and publication productivity in resource-constrained research institutions in a developing country. Res. Pol. 2011; 40 (2):204–216. [ Google Scholar ]
• Zuboff S. Basic Books; NY: 1988. In the Age of the Smart Machine: the Future of Work and Power. [ Google Scholar ]

ORIGINAL RESEARCH article

This article is part of the research topic.

Building the Future of Education Together: Innovation, Complexity, Sustainability, Interdisciplinary Research and Open Science

Developing the Skills for Complex Thinking Research: A Case Study Using Social Robotics to Produce Scientific Papers Provisionally Accepted

• 1 Institute for the Future of Education, Monterrey Institute of Technology and Higher Education (ITESM), Mexico
• 2 University of Cienfuegos, Cuba

The final, formatted version of the article will be published soon.

The development of university students' skills to successfully produce scientific documents has been a recurring topic of study in academia. This paper analyzes the implementation of a training experience using a digital environment mediated by video content materials starring humanoid robots. The research aimed to scale complex thinking and its subcompetencies as a hinge to strengthen basic academic research skills. Students from Colombia, Ecuador, and Mexico committed to preparing a scientific document as part of their professional training participated. A pretest to know their initial level of perception, a posttest to evaluate if there was a change, and a scientific document the students delivered at the end of the training experience comprised the methodology to demonstrate the improvement of their skills. The results indicated students' perceived improvement in the sub-competencies of systemic, creative, scientific, and innovative thinking; however, their perceptions did not align with that of the tutor who reviewed the delivered scientific product. The conclusion was that although the training experience helped strengthen the students' skills, variables that are determinants for a student to develop the knowledge necessary to prepare scientific documents and their derived products remain to be analyzed.

Keywords: higher education, research skills, Educational innovation, complex thinking, scientific thinking, Critical Thinking, Innovative thinking, social robotics

Received: 16 Oct 2023; Accepted: 17 May 2024.

Copyright: © 2024 Lopez-Caudana, George-Reyes and Avello-Martínez. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Dr. Edgar O. Lopez-Caudana, Institute for the Future of Education, Monterrey Institute of Technology and Higher Education (ITESM), Monterrey, Mexico

People also looked at

Research on Agile Project Management with Scrum Method

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

This paper is in the following e-collection/theme issue:

Published on 17.5.2024 in Vol 26 (2024)

Reporting of Ethical Considerations in Qualitative Research Utilizing Social Media Data on Public Health Care: Scoping Review

Authors of this article:

• Yujie Zhang 1, 2 * , MD   ; 
• Jiaqi Fu 1, 2 * , MD   ; 
• Jie Lai 1, 2 * , MD   ; 
• Shisi Deng 1, 2 , MD   ; 
• Zihan Guo 1, 2 , MD   ; 
• Chuhan Zhong 1, 2 , MD   ; 
• Jianyao Tang 1, 2 , MD   ; 
• Wenqiong Cao 1 , BSc   ; 
• Yanni Wu 1 , PhD  

1 Nanfang Hospital, Southern Medical University, Guangzhou, China

2 School of Nursing, Southern Medical University, Guangzhou, China

*these authors contributed equally

Corresponding Author:

Yanni Wu, PhD

Nanfang Hospital

Southern Medical University

No 1838 Guangzhou Avenue North

Baiyun District, Guangdong Province

Guangzhou, 510515

Phone: 86 02061641192

Email: [email protected]

Background: The internet community has become a significant source for researchers to conduct qualitative studies analyzing users’ views, attitudes, and experiences about public health. However, few studies have assessed the ethical issues in qualitative research using social media data.

Objective: This study aims to review the reportage of ethical considerations in qualitative research utilizing social media data on public health care.

Methods: We performed a scoping review of studies mining text from internet communities and published in peer-reviewed journals from 2010 to May 31, 2023. These studies, limited to the English language, were retrieved to evaluate the rates of reporting ethical approval, informed consent, and privacy issues. We searched 5 databases, that is, PubMed, Web of Science, CINAHL, Cochrane, and Embase. Gray literature was supplemented from Google Scholar and OpenGrey websites. Studies using qualitative methods mining text from the internet community focusing on health care topics were deemed eligible. Data extraction was performed using a standardized data extraction spreadsheet. Findings were reported using PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines.

Results: After 4674 titles, abstracts, and full texts were screened, 108 studies on mining text from the internet community were included. Nearly half of the studies were published in the United States, with more studies from 2019 to 2022. Only 59.3% (64/108) of the studies sought ethical approval, 45.3% (49/108) mentioned informed consent, and only 12.9% (14/108) of the studies explicitly obtained informed consent. Approximately 86% (12/14) of the studies that reported informed consent obtained digital informed consent from participants/administrators, while 14% (2/14) did not describe the method used to obtain informed consent. Notably, 70.3% (76/108) of the studies contained users’ written content or posts: 68% (52/76) contained verbatim quotes, while 32% (24/76) paraphrased the quotes to prevent traceability. However, 16% (4/24) of the studies that paraphrased the quotes did not report the paraphrasing methods. Moreover, 18.5% (20/108) of the studies used aggregated data analysis to protect users’ privacy. Furthermore, the rates of reporting ethical approval were different between different countries ( P =.02) and between papers that contained users’ written content (both direct and paraphrased quotes) and papers that did not contain users’ written content ( P <.001).

Conclusions: Our scoping review demonstrates that the reporting of ethical considerations is widely neglected in qualitative research studies using social media data; such studies should be more cautious in citing user quotes to maintain user privacy. Further, our review reveals the need for detailed information on the precautions of obtaining informed consent and paraphrasing to reduce the potential bias. A national consensus of ethical considerations such as ethical approval, informed consent, and privacy issues is needed for qualitative research of health care using social media data of internet communities.

Introduction

Social media are web-based computer-mediated tools to collaborate, share, or exchange information, ideas, pictures, or videos in virtual communities and networks such as message boards, communities, chat rooms, forums, Twitter, and Facebook [ 1 ]. Moreover, patients and researchers can use internet communities to provide health care and disseminate health information [ 2 , 3 ]. Health care refers to the efforts made to improve or maintain physical, mental, or emotional well-being, including prevention, diagnosis, treatment, recovery, and other physical and mental impairments [ 4 ]. Currently, with 57% of the global population’s access to social media, more than 40% of the patients and caregivers worldwide utilize the internet community for health care information needs [ 5 ]. With diverse populations accessing internet communities and sharing information about health care topics, researchers have the opportunity to collect and analyze text about health care from a diverse range of participants in the internet community, which was unavailable previously [ 6 ]. Usually, quantitative data are derived from information extraction, which can be analyzed statistically, and the summary results presented cannot be directly linked to individual participants. In contrast, qualitative research within internet community analysis posts and comments qualitatively or thematically involves a more detailed and in-depth analysis and understanding of the full written content [ 7 ]. However, a controversial ethical problem has been raised about conducting qualitative research containing internet users’ verbatim quotes that could lead to traceability of the original post, thereby causing a threat to an individual’s privacy [ 8 ]. Additionally, a previous study investigated public and patients’ views regarding ethics in research using social media data and reported that internet users were aggrieved if they found any of their quotes cited in a medical research paper without obtaining their informed consent [ 9 ]. Further, besides the privacy breach caused by posts being traced, there is greater harm for special groups or vulnerable groups if we do not highlight the importance of the technical standards for text mining and privacy protection in health care. For instance, some unusual postings, abnormal pictures, and interactions that were expressed by individuals with mental disorders in social media can be detected by researchers by using text mining tools without obtaining their consent [ 10 ]. The publication of research on mental disorders, including quotes in posts, can result in a high risk of information harm, which can lead to personal information being revealed and further stigmatization of the condition or disease [ 11 ]. Since 2001, ethical concerns have been debated for decades about ethical approval, informed consent, and how to ensure anonymity and preserve data privacy and confidentiality in qualitative research in the internet community [ 12 - 14 ].

With the rapid development of social media and internet research, some ethical guidelines or standards have been published to ensure that research based on internet communities is conducted ethically. The Association of Internet Researchers (internet research ethical guidelines 2.0 and 3.0) showed that researchers working without the direct approval of ethics review boards would have additional challenges to face, and obtaining informed consent is obviously impracticable in several big data projects. However, with the ethical issues about privacy breaches and harms of risk of discrimination, the Association of Internet Researchers recommended reserving the acquisition of informed consent to the dissemination stage by asking for informed consent from specific participants before publication of their quotes [ 15 , 16 ]. Furthermore, researchers should take responsibility for information confidentiality and anonymity according to the internet research ethics criteria prepared by the National Committee for Research Ethics in the Social Sciences and the Humanities guidelines, which recommend a basic research ethic norm for the analyses, reports, and evaluations that apply to all research [ 17 ]. Moreover, the National Committee for Research Ethics in the Social Sciences and the Humanities guidelines contain more details about the demand for legal consent and privacy standards imposed by the European Union’s General Data Protection Regulation. The General Data Protection Regulation is a European Union–wide regulation targeting the project of personal data processing. The General Data Protection Regulation defines personal data as any information relating to an identifiable person (data subject), including name, online identification number, location data, and other factors related to personal, physical, physiological, mental, or social identity [ 18 ]. The General Data Protection Regulation recommends using anonymous data and deleting identifiable information to ensure the confidentiality of the data. Consent should be obtained from the individual for use in scientific research [ 18 , 19 ]. The British Psychological Society guideline does not explicitly refer to the internet community but suggests that researchers may consider paraphrasing the verbatim quotes to reduce the risk of being traced or identified in qualitative research [ 20 ]. When paraphrasing, steps must be put into place to ensure that the original meaning of the message is maintained. Currently, there is no widespread consensus on ethical considerations by social media researchers.

Some researchers have tried to explore the reporting of existing ethical considerations in research papers using social media data. For instance, Sinnenberg et al [ 6 ] reported that only 32% and 12% of the papers mentioned acquiring ethical approval and informed consent, respectively, by utilizing multiple analysis methods, including surveillance, intervention, recruitment, engagement, content analysis, and network analysis with Twitter data before 2015. Thereafter, Takats et al [ 21 ] conducted an updated examination based on Sinnenberg et al’s [ 6 ] study. They found that of 367 studies using different methodological approaches, including sentiment mining, surveillance, and thematic exploration of public health research using Twitter data between 2010 to 2019, 17% of the studies included verbatim tweets and identifiable information about the internet users [ 21 ]. Similarly, Lathan et al [ 22 ] reviewed papers, including both qualitative and quantitative methods, by using Facebook data to explore public health issues and reported that only 48% and 10% of the papers obtained ethical approval and informed consent, respectively. Furthermore, in a study on research using YouTube data or comments, Tanner et al [ 23 ] found that only 26.1% of these studies sought ethical approval, only 1 paper (0.08%) sought informed consent, and 27.7% contained identifiable information. These findings indicate widespread neglect of ethical issues such as ethical approval, informed consent, and privacy issues in research papers using social media data.

Our study focuses on the ethical challenges of qualitative studies utilizing social media data. First, social media can be considered as sources for qualitative data collection because of the low cost, vast amount of available sources about health information, and users’ health behaviors, experiences, and attitudes. Second, qualitative research is context-dependent and mainly contains quotations and written content to support the viewpoint. It is acknowledged that quote materials from social media would potentially be traced back to the original posts and threaten the users’ privacy [ 24 ]. This is supported by findings reported by Ayers et al [ 25 ] who found that online searches of verbatim Twitter quotes in journal papers described as “content analyses” or “coded Twitter postings” can be traced back to individual internet users 84% of the time. Furthermore, Lathan et al [ 22 ] identified that 46% of the studies with verbatim or paraphrased quotes could be traced to the original posts in 10 minutes. Therefore, it is essential to investigate the extent to which ethical oversight is reported in qualitative studies using social media data. Moreover, qualitative research often involves personally sensitive data about health conditions and diseases; hence, anonymity and proper deidentification would be more important for researchers [ 26 , 27 ].

Previous studies have reviewed the ethical challenges and methodological use of social media platforms such as Twitter [ 6 , 21 ], Facebook [ 22 ], and YouTube [ 23 ] for health care research in both qualitative and quantitative studies. Although there is plenty of qualitative data pouring into social media such as blogs, Twitter, Facebook, and Weibo, evidence is lacking on the investigation of ethical considerations targeting qualitative data in different software and web-based discussion forums to provide a more comprehensive understanding of the ethical issues. To address the ethical considerations in qualitative research of different internet communities and draw the attention of researchers and publishers to ethical issues, we conducted this study to evaluate the ethical practices and ethical considerations of qualitative studies on health care by using data of internet communities. This review aims to (1) assess the rates of reporting institutional review board (IRB) approval and informed consent in studies focused on mining text in the internet community and social media, (2) compare these rates according to the year of publication, country conducting the research, website included in the study’s analysis, and journal’s guidelines about ethical approval for the type of study, and (3) describe whether the studies used anonymized/deidentified data.

Research Design

We conducted a scoping review to investigate how qualitative research mining social media data handles ethical approval, informed consent, and confidential issues. We performed this study according to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. The completed PRISMA-ScR checklist is provided in Multimedia Appendix 1 .

Search Strategy

All published qualitative studies from 2010 to March 31, 2023, focusing on mining text from online community and social media sources about health care in the following databases were included in this study: PubMed, Web of Science, CINAHL, Cochrane, and Embase. A standardized search string containing Medical Subject Headings (MeSH) and non-MeSH entry terms was used in the search strategy. In addition, the reference lists of the retrieved papers and citation tracking were manually searched as a supplement to database searches to improve comprehensiveness. Gray literature was also identified through internet searches in Google Scholar and OpenGrey websites. The search strategies are represented in Multimedia Appendix 2 .

Inclusion and Exclusion Criteria

We divided the criteria into 2 parts. First, we limited the inclusion and exclusion criteria used at the title and abstract screening stage eligible for (1) studies mining existing text and posts from the internet community and social media data focusing on health care topics, (2) studies using qualitative methods or available qualitative parts in mixed methods studies to analyze data, and (3) studies only written in English. Ineligible studies were those related to investigating the use and dissemination of social media in health care, using social media or internet community as an intervention tool, and using social media to conduct web-based interviews, surveys, or focus groups. We also excluded studies published as reviews, case studies, conference abstracts, commentaries, policies, guidelines, and recommendations. Second, at the full screening stage, the specific eligible inclusion criteria were studies focused on mining text about health care topics with full-text papers. Studies that did not have the full text after contacting the authors and that were not originally in the English language were excluded.

Study Selection

All results of the searches were entered into the EndNote library, and duplicates were removed. Two researchers reviewed the titles and abstracts based on the inclusion and exclusion criteria independently. Those studies that were irrelevant to the study topic were discarded, and then the full text was screened to select eligible papers. Any disagreements were discussed and resolved by consensus or a third person.

Data Extraction

Data were extracted between April 2023 and May 2023. Two researchers independently read the full text carefully, and the results were extracted using a standardized data extraction spreadsheet, including research type, first author, study objective, sample size, publication time, country where the research was conducted or country of the first author, website or internet community the studies focus on, type of data collected from social media, language of collected posts or data, privacy level of data (public or privacy posts), study design, research results, published journal, and information about the ethical considerations. Disagreements were resolved by consensus of a third person. The information about ethical considerations was analyzed to investigate the rates of reporting ethical approval, informed consent, and privacy issues: whether IRB review was reported (IRB approval, IRB exemption, unnecessary, not mentioned) and the reason for not requiring IRB approval; whether informed consent was obtained from participants or the websites’ administrators, consent types (digitally informed consent or written informed consent, informed consent is not required, consent was waived by IRB), and the methods used to obtain consent in each study; and whether quoting a post in papers could lead to the identification of internet users in each study. The description of users’ posts (verbatim quote, paraphrase) was recorded. We also analyzed if posts were paraphrased to maintain the original meaning, if actions were taken to deidentify the internet users, and if the posts contained other identifying information (ie, usernames, photos, links, hashtags) attached to the post. As every journal would provide publication ethical considerations and requirements, we also searched the submission guidelines and editorial policies of each journal submission website to check whether the journal contained any ethical guidance targeting studies using data from internet community and social media platforms. Additional information was included about the details of ethical approval, informed consent, and privacy, for example, whether individuals can withdraw their quotes if they want to be excluded from the study at any time without any reprisal and whether the quotations were tested for deidentification via search engines. There was excellent agreement on the primary outcome between the 2 researchers (k>.95 for all).

Data Analysis

Data were analyzed using SPSS software (IBM Corp). The chi-square test or Fisher exact tests (when cell size was less than 5) were used to test for differences between the rates of informed consent and ethical approval according to publication year, website, and different countries. All P values were 2-sided, and P values <.05 indicated significance.

Study Selection for the Review

We reviewed 4674 papers after removing the duplicates. After screening the titles, abstracts, and full-texts, we reviewed 108 eligible papers ( Figure 1 ). The full list of the included papers and all the extracted information are incorporated in Multimedia Appendix 3 [ 28 - 135 ]. Of the 108 studies reviewed, 73 (67.6%) were qualitative studies and 35 (32.4%) were mixed methods studies. All papers had text mined from internet communities or social media for qualitative analysis. The sample size ranged from 32 to 392,962. Approximately 82.4% (89/108) of the studies were published after 2018, and there was a sharp increase in the number of studies from 2019 to 2022. Moreover, nearly half of the studies (55/108, 50.9%) were published in the United States. Regarding the websites for mining text, the most widely used social media platform was Twitter (42/108, 38.9%), followed by Facebook (17/108, 15.7%).

Ethics Approval in These Studies

Our results indicated that of the 108 studies, 78 (72.2%) reported ethics approval. Of the 78 studies, 31 (40%) explicitly stated that ethics approval was obtained before the study was undertaken, 33 (42%) reported that the ethics approval was granted through exemptions by the local IRB, and 14 (18%) explicitly demonstrated that approval by the ethics committee was not required because publicly available data were collected from internet communities and social media platforms. However, 30 (27.8%) of the 108 studies did not mention about obtaining IRB approval ( Table 1 ).

Based on our exploration of the ethical guidelines of each journal to determine whether there were ethical requirements for studies mining social media data, only 36.1% (39/108) of the studies were published in journals that required ethical considerations for studies gathering data from social media platforms by using internet and digital technologies. Of the 39 studies published in 19 journals, 27 (69%) were published in the Journal of Medical Internet Research and its sister journals. The submission guidelines of the Journal of Medical Internet Research state that authors of manuscripts describing studies of internet, digital tools, and technologies are required to verify that they have adhered to local, national, regional, and international laws and regulations, and are required to verify that they complied with informed consent guidelines. Moreover, 2 journals also provided a specific requirement, that is, when researchers interact with individuals or obtain privacy information gathered from social media platforms, they should obtain ethics approval prior to conducting the study and informed consent from anyone who could potentially be identified. Surprisingly, there were no significant differences in the ethics approval reportage between journals with ethics approval guidelines and those that did not have ethics guidelines for researchers gathering data from social media platforms ( P =.08). Notably, the rates of reporting ethics approval were different between different countries ( P =.02). However, there were no statistically significant differences between the rates of reporting ethical approval and different websites or publication years (all P >.05) ( Table 2 ).

Informed Consent

Of the 108 studies, 59 (54.7%) showed that they did not include any information about informed consent and 49 (45.3%) mentioned informed consent. Of the 49 studies that mentioned informed consent, 14 (13%) demonstrated that informed consent was waived by local institutional boards, and 21 (19%) reported that informed consent was not required because this information is publicly available in websites or did not involve human participants. We interpreted this as not seeking informed consent. Only 14 (12.9%) of the 108 studies explicitly indicated that informed consent was obtained ( Table 1 ). Among the 14 studies, 2 (14%) only provided a generic statement that informed consent was obtained but did not report the process of how the informed consent was obtained, while 12 (86%) received digital informed consent. Of the 12 studies that reported receiving digital informed consent, 6 reported that they sought permission from the communities’ or groups’ administrators and by posting a statement of the research objective on the group’s wall, while 5 studies contacted the participants privately via email, commenting below the posts and software to gain consent, and 1 study reported that it had sent a digital version of the informed consent book. Furthermore, among the studies that had obtained informed consent, 7 studies included the statement that the individuals’ posts would be removed if they wanted to be excluded from the study, and they could withdraw from the study whenever they wanted. In addition, the rates of reporting informed consent showed no statistical significance between publication years, different countries, and different websites (all P >.05) ( Table 2 ).

Confidentiality of the Information

All data sources were obtained from anonymous websites or communities, and the majority (104/108, 96.3%) of the data sources did not contain usernames. Notably, only 3.7% (4/108) of the studies contained the participants’ usernames or pseudonyms. One study reported that pseudonyms like Sasha had been used instead of the real name. The other 3 studies contained the expression for usernames but did not state whether pseudonyms were used. Except for 9 studies that used nonnative language quotes and 3 studies that were transcribed into text via video, among the 108 included studies, 76 (70.3%) quoted at least one native language post in their reports. Additionally, 20 studies presenting aggregated analysis or composite accounts did not include any quotation or written content. Of the 76 studies containing internet users’ written content, 52 (68%) contained just verbatim-quoted participants’ posts and 24 (32%) contained paraphrased posts ( Table 1 ). Among the 52 studies containing direct and verbatim quotations, which are likely to be traced to the original posts from users, only 17 (33%) studies took measures to deidentify the users. The 17 studies mentioned that all names or usernames were removed and personal identifying information was removed to maintain privacy, while 42% (22/52) of the studies did not mention any measures that were taken to deidentify the users and maintain confidentiality. Approximately 32% (24/76) of the studies described that they paraphrased posts and removed any explicitly identified personal information to maintain confidentiality to reduce the likelihood of users being identified via search engines. Of the 24 studies, 20 (83%) reported that the quotations were slightly modified or summarized for readability, the symbol information was removed using “…”, and key identifiable information was removed to protect privacy while maintaining the meaning of posts. Four of the 24 (17%) studies did not report the methods and details of paraphrasing. Notably, only 3% (2/76) of the studies containing users’ written content showed that researchers intentionally entered each quote into search engines to ensure that every quote did not lead to the original posts. Moreover, of the 76 studies containing written content, 62 (82%) did not contain other types of identity information attached to the posts, while 14 (18%) included other identifying data (hashtags, emojis, geolocation, photos, links, screenshots) attached to the original posts for analysis ( Table 1 ). Of the 14 studies including other identifying information, 4 (29%) contained photos and screenshots associated with the website pages. Of the 52 studies that disclosed verbatim quotes and other identifiable information, 26 (46%) studies reported informed consent consideration, and only 8 (15%) obtained explicitly informed consent. Additionally, of the 77% (40/52) of the studies that mentioned IRB or ethical review, 38% (15/40) received IRB approval, and 63% (25/40) of the studies were granted exemption. The proportion of reporting ethical approval in studies containing users’ written content was modestly higher than that in studies not containing users’ written content (60/76, 79% vs 14/32, 44%; P <.001) ( Table 2 ).

Principal Findings and Comparison to Prior Work

In this scoping review, we included 108 studies ( Multimedia Appendix 3 ; [ 28 - 135 ]) that focused on mining text from internet community and social media data for health care research, and we reviewed the ethical consideration reportage and outcome reports in these studies. We found that the rates of reporting IRB approval and informed consent in qualitative research on health care utilizing social media data were 59.3% (64/108) and 12.9% (14/108), respectively. Our findings demonstrate that the key ethical considerations for qualitative research in online communities are insufficiently discussed and described. However, the reporting rates of ethical considerations in the papers in our scoping review were much higher than those reported in systematic reviews including multiple analysis methodologies on only 1 social media platform. For example, ethics approval and informed consent were reported in 48% and 10% of research studies using only Facebook data [ 22 ], 32% and 0% from 2006 to 2019 [ 21 ], 40% and 0.9% (only 1 paper) from 2015 to 2016 in public health research using only Twitter data [ 25 ], and 26.1% and 0.8% (only 1 paper) in researches incorporating only YouTube data [ 23 ], respectively. In fact, previous studies were limited to only a few selected websites such as Twitter, Facebook, and YouTube. There is a lack of research that incorporates a variety of different social media data for comparisons. Differences in the reporting of ethical considerations may be attributed to the different methodologies adopted by studies. For example, Lathan et al [ 22 ] analyzed the ethical considerations in studies including predictive or model development, while our research focuses on the ethical considerations in qualitative studies.

Importantly, our findings indicate that there is a need to develop a standardized and apparent approach for the reporting of ethical considerations in qualitative research of data from social media and online communities. Our research demonstrates that the rates of reporting ethics approval are different in different countries ( P =.02). Specifically, a wide variety of national research ethics governing bodies and over 1000 laws, regulations, and standards provide oversight for human subjects research in 130 countries. Obviously, a guideline is needed for best ethical practices for qualitative research involving posts from social media platforms. Surprisingly, there were no significant differences between the rates of reporting ethical approval and those of journals specifying ethical requirements for studies involving text mining ( P =.08). This inconsistent result of publication guidelines and reports of ethical approval consent is similar to previous findings on the ethical standards in COVID-19 human studies [ 136 ]. Although there are journal publication guidelines for studies mining social media data, the reports of ethical approval and consent in the papers published in such journals do not exactly follow the guidelines. Consequently, this finding indicates that more ethical awareness is needed among researchers, editors, and reviewers for qualitative studies on data mining.

Besides the different legal and regulations in different countries, the inconsistency in the ethics approval in published papers may be because social media research is a highly interdisciplinary science, and computer science researchers may be less experienced or may pay less attention to the key ethical issues of protecting human subjects [ 137 ]. Medical and health science researchers may have considered some ethical concerns about gathering social media data but they may not be familiar with the relevant guidelines. For example, the Association of Internet Researchers has a detailed ethical guideline targeting social scientists conducting digital research, while it may be less popular and less well-known among medical and health care researchers. At the institute level, Ferretti et al [ 138 ] noticed that institutionalized review committees, especially the individual IRB institutes for universities and health care systems lack knowledge about the methodology, text mining technical standards, data security, and ethical harms for studies using big data and social media as sources. Because of this lack of knowledge, institutional ethics committees may have inconsistent ethical criteria and perspectives about web-based projects using social media data [ 139 ]. Therefore, some ethics review committees exclude research on internet communities from ethical oversights because their ethics standards are confined only to medical fields. Above all, it is additionally challenging for ethical approval institutions because of the continuous development and dynamic change of studies using social media data. Furthermore, it is necessary for ethics committee members to be trained about the ethical issues in studies mining text from social media. Inviting interdisciplinary researchers to join in the approval process would be an appropriate method to increase the awareness of ethical considerations [ 140 , 141 ].

Interestingly, the reporting rate of obtaining informed consent for mining social media data in qualitative studies was unexpected. The most influential ethical reports such as the Nuremberg Code [ 142 ], Declaration of Helsinki [ 143 ], and the Belmont Report [ 144 ] have demonstrated the principle of informed consent in research involving humans. Our review shows that only 12.9% (14/108) of the studies explicitly obtained informed consent and 32.4% (35/108) of the studies reported that informed consent was exempted by IRB or was not required, as the information was available publicly in websites or did not involve human participants. Our results are similar to those of Wongkoblap et al [ 145 ] who reported that only 16.7% of the studies received informed consent from participants prior to data analysis on data mining of social network data on mental health disorders.

There are multiple reasons for the challenges in obtaining informed consent in an internet setting. First, it is impractical for researchers to gain individual informed consent from a large number of users in an internet community [ 146 ]. Second, members of ethics review boards lack consensus about the need for informed consent from an internet community for qualitative research under the current legal definition [ 147 ]. Moreover, there has been a debate on the criteria of human subject research in using social media data. The federal regulation recommends that if data in the studies are obtained from public social media websites, where data are identifiable and do not require interaction with individuals, such studies do not constitute human subject research, while studies involving the identification of private information or interaction with the individual can be considered as human subject research [ 148 ]. In contrast, some researchers believe that social media and big data research are not ethically exceptional and should be treated in the same manner and with the same rules as those for traditional forms of research [ 149 ]. There is ambiguity as to what is appropriate or should be standard practice for obtaining informed consent.

Currently, it is challenging to maintain privacy and protect the traceability of individuals posting content in the internet community. Our findings indicated that 70.3% (76/108) of the studies contained internet users’ written content, of which 68% (52/76) included verbatim quotations of users’ posts that could lead to identification, and 18% (14/76) of the studies included other identifiable information such as links, screenshots, and emojis linked to original posts, which are similar to the findings of Ayers et al [ 25 ] and Lathan et al [ 22 ]. Usha Lawrance et al [ 150 ] and Wilkinson and Thelwall [ 151 ] argued that using direct quotes to support findings would lead to the identification of users and breach users’ confidentiality in internet community data. Moreover, quoting social media posts or disclosing usernames violate the International Committee of Medical Journal Editors’ ethics standards, which state that identifying information such as written descriptions and photos should not be published unless the information is essential for scientific purposes and the participants give written informed consent for publication [ 152 ]. Furthermore, our study demonstrates that the proportion of studies containing users’ written content (both direct and paraphrased quotations) is higher than that of studies that do not include any quotation or written content (60/76, 79% vs 14/32, 44%; P <.001)——a tentative explanation is that some researchers realized that ethical reportage should be stricter for qualitative papers with quotations from social media posts due to privacy and security issues. This is supported by Boyd and Crawford [ 153 ] who stated that rigorous thinking about the process of mining and anonymizing big data is required for ethics boards to ensure that people are protected. Our findings show that 32% (24/76) of the studies intentionally paraphrased the quotes to ensure that users could not locate them, and 20 studies used aggregated data interconnected with anonymity. Moreover, it is recommended by Wilkinson and Thelwall [ 151 ], Bond et al [ 154 ], and Markham et al [ 155 ] that researchers should not directly quote and work with aggregate data sets and separate texts from their original context, which is more acceptable to participants. In addition, the British Psychological Society guidelines recommend that researchers consider paraphrasing any verbatim quotes to reduce the risk of these being traced to the source [ 20 ]. Notably, 13 of the 25 papers in this study showed that they did not report the precautions taken for paraphrasing. This may be due to the lack of detailed methodology and consensus about paraphrasing quotes to reduce bias and maintaining the original meaning.

Limitations and Strengths

Our scoping review has several limitations. First, our research was limited to qualitative studies and the qualitative parts in mixed methods studies on text mining from social media, and it is unclear whether ethical considerations are critical in quantitative studies among internet communities. Second, we were restricted to studies published in English language and those with the full text available, and therefore, we could be underestimating the number of relevant papers published in other languages. Third, the rates of reporting ethical approval, informed consent, and privacy of this research relied on self-reported data. Thus, it is possible that although certain studies did not report the process of ethical considerations, such considerations may have been followed during the research. Conversely, some studies may have mentioned about the ethical considerations but may not have conducted them in practice. Hence, there is a bias because of the lack of accurate documentation that must be considered.

Social media text mining can be a useful tool for researchers to understand patient experiences of health conditions and health care. However, as illustrated by the absence of ethical discourse in publications, our analysis indicates significant gaps in the ethical considerations and governance of qualitative research of internet posts. Therefore, a complete and consistent consensus guideline of ethical considerations in qualitative research of internet posts is needed to protect users’ data. With the continued advancing development of text-mining techniques, qualitative studies mining text from social media should be more cautious while using user quotations to maintain user privacy and protect the traceability of the internet users posting content. We suggest that authors should report their results by using aggregated findings or deidentified ways like paraphrasing instead of verbatim quotations, which can prevent internet users from being identified through search engines. In addition, authors should provide more detailed information about the precautions taken for obtaining informed consent and paraphrasing to reduce the potential bias. Furthermore, journals and editors should pay more attention to the reporting standards of ethical consideration and privacy issues in qualitative research involving social media data.

Acknowledgments

This project was funded by the National Natural Science Foundation of China (72304131) and the Outstanding Youths Development Scheme of Nanfang Hospital, Southern Medical University (2023J005). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of this manuscript. We sincerely thank the funders of this study.

Data Availability

All data extracted and analyzed during this study are presented in this paper and in the multimedia appendices.

Authors' Contributions

YW was responsible for the protocol of the research and redrafted the paper critically. YZ and JF performed literature searches. YZ, JL, and WC performed study identification and screening. ZG, SD, CZ, and JT extracted and analyzed the data from the included journals. YZ and JL wrote the first draft of the paper. All authors read and approved the final manuscript.

Conflicts of Interest

None declared.

PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) checklist.

Search strategy for each database.

Summary of included literature.

• Obar JA, Wildman S. Social media definition and the governance challenge: An introduction to the special issue. Telecommunications Policy. Oct 2015;39(9):745-750. [ CrossRef ]
• Kaplan AM, Haenlein M. Users of the world, unite! The challenges and opportunities of social media. Business Horizons. Jan 2010;53(1):59-68. [ CrossRef ]
• VanDam C, Kanthawala S, Pratt W, Chai J, Huh J. Detecting clinically related content in online patient posts. J Biomed Inform. Nov 2017;75:96-106. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Health care. Wikipedia. URL: https://en.wikipedia.org/wiki/Health_care [accessed 2023-06-01]
• Moorhead SA, Hazlett DE, Harrison L, Carroll JK, Irwin A, Hoving C. A new dimension of health care: systematic review of the uses, benefits, and limitations of social media for health communication. J Med Internet Res. Apr 23, 2013;15(4):e85. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sinnenberg L, Buttenheim AM, Padrez K, Mancheno C, Ungar L, Merchant RM. Twitter as a tool for health research: A systematic review. Am J Public Health. Jan 2017;107(1):e1-e8. [ CrossRef ] [ Medline ]
• Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. Jan 2006;3(2):77-101. [ CrossRef ]
• Hswen Y, Naslund JA, Brownstein JS, Hawkins JB. Monitoring online discussions about suicide among Twitter users with schizophrenia: exploratory study. JMIR Ment Health. Dec 13, 2018;5(4):e11483. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Golder S, Ahmed S, Norman G, Booth A. Attitudes toward the ethics of research using social media: a systematic review. J Med Internet Res. Jun 06, 2017;19(6):e195. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Athanasopoulou C, Sakellari E. Facebook and health information: content analysis of groups related to schizophrenia. Stud Health Technol Inform. 2015;213:255-258. [ Medline ]
• Conway M, O'Connor D. Social media, big data, and mental health: current advances and ethical implications. Curr Opin Psychol. Jun 2016;9:77-82. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Eysenbach G, Till JE. Ethical issues in qualitative research on internet communities. BMJ. Nov 10, 2001;323(7321):1103-1105. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Heilferty CM. Ethical considerations in the study of online illness narratives: a qualitative review. J Adv Nurs. May 2011;67(5):945-953. [ CrossRef ] [ Medline ]
• Roberts LD. Ethical issues in conducting qualitative research in online communities. Qualitative Research in Psychology. Apr 20, 2015;12(3):314-325. [ CrossRef ]
• Franzke A, Bechmann A, Zimmer M, Ess C, the Association of Internet Researchers. Internet research: ethical guidelines 3.0. Association of Internet Researchers. 2020. URL: https://aoir.org/reports/ethics3.pdf [accessed 2023-05-30]
• Markham A, Buchanan E. Ethical decision-making and internet research: recommendations from the AoIR ethics working committee (version 2.0). AoIR. 2012. URL: https://aoir.org/reports/ethics2.pdf [accessed 2023-06-05]
• The NCFREITSSH(. Ethical guidelines for internet research. 2022. URL: https:/​/www.​forskningsetikk.no/​en/​guidelines/​social-sciences-humanities-law-and-theology/​a-guide-to-internet-research-ethics [accessed 2023-06-13]
• Chapter 3: Rights of the data subject. General Data Protection Regulation (GDPR). European Commission. Official Journal of the European Union; 2018. URL: https://gdpr-info.eu/chapter-3/ [accessed 2023-11-29]
• Hand DJ. Aspects of data ethics in a changing world: where are we now? Big Data. Sep 01, 2018;6(3):176-190. [ FREE Full text ] [ CrossRef ] [ Medline ]
• British Psychological Society. Ethics guidelines for internet-mediated research. 2021. URL: https:/​/explore.​bps.org.uk/​binary/​bpsworks/​64374754e0c1dd30/​5a757d7c2d39a9837f0eedec1b9ba28fa5e9e38f0bffa9950b678ab727803959/​rep155_2021.​pdf [accessed 2023-05-30]
• Takats C, Kwan A, Wormer R, Goldman D, Jones HE, Romero D. Ethical and methodological considerations of Twitter data for public health research: systematic review. J Med Internet Res. Nov 29, 2022;24(11):e40380. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lathan HS, Kwan A, Takats C, Tanner JP, Wormer R, Romero D, et al. Ethical considerations and methodological uses of Facebook data in public health research: A systematic review. Soc Sci Med. Apr 2023;322:115807. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tanner J, Takats C, Lathan H, Kwan A, Wormer R, Romero D, et al. Approaches to research ethics in health research on YouTube: systematic review. J Med Internet Res. Oct 04, 2023;25:e43060. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stockdale J, Cassell J, Ford E. "Giving something back": A systematic review and ethical enquiry into public views on the use of patient data for research in the United Kingdom and the Republic of Ireland. Wellcome Open Res. 2018;3:6. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ayers JW, Caputi TL, Nebeker C, Dredze M. Don't quote me: reverse identification of research participants in social media studies. NPJ Digit Med. 2018;1:30. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Aitken M, de St Jorre J, Pagliari C, Jepson R, Cunningham-Burley S. Public responses to the sharing and linkage of health data for research purposes: a systematic review and thematic synthesis of qualitative studies. BMC Med Ethics. Nov 10, 2016;17(1):73. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hunter RF, Gough A, O’Kane N, McKeown G, Fitzpatrick A, Walker T, et al. Ethical issues in social media research for public health. Am J Public Health. Mar 2018;108(3):343-348. [ CrossRef ]
• Ahmed OH, Sullivan SJ, Schneiders AG, McCrory P. iSupport: do social networking sites have a role to play in concussion awareness? Disabil Rehabil. 2010;32(22):1877-1883. [ CrossRef ] [ Medline ]
• Bender JL, Jimenez-Marroquin M, Jadad AR. Seeking support on Facebook: a content analysis of breast cancer groups. J Med Internet Res. Feb 04, 2011;13(1):e16. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gajaria A, Yeung E, Goodale T, Charach A. Beliefs about attention-deficit/hyperactivity disorder and response to stereotypes: youth postings in Facebook groups. J Adolesc Health. Jul 2011;49(1):15-20. [ CrossRef ] [ Medline ]
• Degroot JM. Maintaining relational continuity with the deceased on Facebook. Omega (Westport). Nov 01, 2012;65(3):195-212. [ CrossRef ]
• Donelle L, Booth RG. Health tweets: an exploration of health promotion on twitter. Online J Issues Nurs. Sep 30, 2012;17(3):4. [ FREE Full text ] [ Medline ]
• Rui JR, Chen Y, Damiano A. Health organizations providing and seeking social support: a Twitter-based content analysis. Cyberpsychol Behav Soc Netw. Sep 2013;16(9):669-673. [ CrossRef ] [ Medline ]
• Lyles CR, López A, Pasick R, Sarkar U. "5 mins of uncomfyness is better than dealing with cancer 4 a lifetime": an exploratory qualitative analysis of cervical and breast cancer screening dialogue on Twitter. J Cancer Educ. Mar 2013;28(1):127-133. [ CrossRef ] [ Medline ]
• Lee JL, DeCamp M, Dredze M, Chisolm MS, Berger ZD. What are health-related users tweeting? A qualitative content analysis of health-related users and their messages on twitter. J Med Internet Res. Oct 15, 2014;16(10):e237. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ahlwardt K, Heaivilin N, Gibbs J, Page J, Gerbert B, Tsoh JY. Tweeting about pain: comparing self-reported toothache experiences with those of backaches, earaches and headaches. J Am Dent Assoc. Jul 2014;145(7):737-743. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Struik LL, Baskerville NB. The role of Facebook in Crush the Crave, a mobile- and social media-based smoking cessation intervention: qualitative framework analysis of posts. J Med Internet Res. Jul 11, 2014;16(7):e170. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Abdel-Razig S, Anglade P, Ibrahim H. Impact of the COVID-19 pandemic on a physician group's WhatsApp chat: qualitative content analysis. JMIR Form Res. Dec 07, 2021;5(12):e31791. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Reddy A. Skincare in social media: analyzing prominent themes in online dermatologic discussions. Cureus. May 7, 2021:e14890. [ CrossRef ] [ Medline ]
• van der Pijl MSG, Hollander MH, van der Linden T, Verweij R, Holten L, Kingma E, et al. Left powerless: A qualitative social media content analysis of the Dutch #breakthesilence campaign on negative and traumatic experiences of labour and birth. PLoS One. 2020;15(5):e0233114. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rath L, Vijiaratnam N, Skibina O. Alemtuzumab in multiple sclerosis: lessons from social media in enhancing patient care. Int J MS Care. 2017;19(6):323-328. [ CrossRef ] [ Medline ]
• Bloom R, Beck S, Chou W, Reblin M, Ellington L. In their own words: experiences of caregivers of adults with cancer as expressed on social media. ONF. Sep 1, 2019;46(5):617-630. [ CrossRef ]
• Grumme V, Gordon S. Social media use by transplant recipients for support and healing. Comput Inform Nurs. Dec 2016;34(12):570-577. [ CrossRef ] [ Medline ]
• Haug NA, Bielenberg J, Linder SH, Lembke A. Assessment of provider attitudes toward #naloxone on Twitter. Subst Abus. 2016;37(1):35-41. [ CrossRef ] [ Medline ]
• Oren E, Martinez L, Hensley RE, Jain P, Ahmed T, Purnajo I, et al. Twitter communication during an outbreak of hepatitis A in San Diego, 2016–2018. Am J Public Health. Oct 2020;110(S3):S348-S355. [ CrossRef ]
• Parker C, Zomer E, Liew D, Ayton D. Characterising experiences with acute myeloid leukaemia using an Instagram content analysis. PLoS One. 2021;16(5):e0250641. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kurko T, Linden K, Kolstela M, Pietilä K, Airaksinen M. Is nicotine replacement therapy overvalued in smoking cessation? Analysis of smokers' and quitters' communication in social media. Health Expect. Dec 2015;18(6):2962-2977. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Reuter K, Lee D. Perspectives toward seeking treatment among patients with psoriasis: protocol for a Twitter content analysis. JMIR Res Protoc. Feb 18, 2021;10(2):e13731. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mollema L, Harmsen IA, Broekhuizen E, Clijnk R, De Melker H, Paulussen T, et al. Disease detection or public opinion reflection? Content analysis of tweets, other social media, and online newspapers during the measles outbreak in The Netherlands in 2013. J Med Internet Res. May 26, 2015;17(5):e128. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Hagan ET, Traeger AC, Bunzli S, Leake HB, Schabrun SM, Wand BM, et al. What do people post on social media relative to low back pain? A content analysis of Australian data. Musculoskelet Sci Pract. Aug 2021;54:102402. [ CrossRef ] [ Medline ]
• Zhou F, Zhang W, Cai H, Cao Y. Portrayals of 2v, 4v and 9vHPV vaccines on Chinese social media: a content analysis of hot posts on Sina Weibo. Hum Vaccin Immunother. Nov 02, 2021;17(11):4433-4441. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Golder S, Bach M, O'Connor K, Gross R, Hennessy S, Gonzalez Hernandez G. Public perspectives on anti-diabetic drugs: exploratory analysis of Twitter posts. JMIR Diabetes. Jan 26, 2021;6(1):e24681. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mercier R, Senter K, Webster R, Henderson RA. Instagram users? experiences of miscarriage. Obstet Gynecol 2020 Jan;? Jan 2020;135(1):166-173. [ CrossRef ]
• Alghamdi A, Abumelha K, Allarakia J, Al-Shehri A. Conversations and misconceptions about chemotherapy in Arabic tweets: content analysis. J Med Internet Res. Jul 29, 2020;22(7):e13979. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Golder S, O'Connor K, Hennessy S, Gross R, Gonzalez-Hernandez G. Assessment of beliefs and attitudes about statins posted on Twitter: a qualitative study. JAMA Netw Open. Jun 01, 2020;3(6):e208953. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stekelenburg N, Horsham C, O'Hara M, Janda M. Using social media to determine the affective and cognitive components of tweets about sunburn. Dermatology. Feb 27, 2020;236(2):75-80. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Meeking K. Patients' experiences of radiotherapy: Insights from Twitter. Radiography (Lond). Aug 2020;26(3):e146-e151. [ CrossRef ] [ Medline ]
• Shah S, Bradbury-Jones C, Taylor J. Using Facebook to tell stories of premature ageing and sexual and reproductive healthcare across the life course for women with cerebral palsy in the UK and USA. BMJ Open. Feb 17, 2020;10(2):e032172. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pretorius K, Choi E, Kang S, Mackert M. Sudden infant death syndrome on Facebook: qualitative descriptive content analysis to guide prevention efforts. J Med Internet Res. Jul 30, 2020;22(7):e18474. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Karmegam D, Mapillairaju B. What people share about the COVID-19 outbreak on Twitter? An exploratory analysis. BMJ Health Care Inform. Nov 2020;27(3):e100133. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hairston TK, Links AR, Harris V, Tunkel DE, Walsh J, Beach MC, et al. Evaluation of parental perspectives and concerns about pediatric tonsillectomy in social media. JAMA Otolaryngol Head Neck Surg. Jan 01, 2019;145(1):45-52. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Årsand E, Bradway M, Gabarron E. What are diabetes patients versus health care personnel discussing on social media? J Diabetes Sci Technol. Mar 2019;13(2):198-205. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jiang X, Jiang W, Cai J, Su Q, Zhou Z, He L, et al. Characterizing media content and effects of organ donation on a social media platform: content analysis. J Med Internet Res. Mar 12, 2019;21(3):e13058. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Oser TK, Minnehan KA, Wong G, Parascando J, McGinley E, Radico J, et al. Using social media to broaden understanding of the barriers and facilitators to exercise in adults with type 1 diabetes. J Diabetes Sci Technol. May 2019;13(3):457-465. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sutton J, Vos SC, Olson MK, Woods C, Cohen E, Gibson CB, et al. Lung cancer messages on Twitter: content analysis and evaluation. J Am Coll Radiol. Jan 2018;15(1 Pt B):210-217. [ CrossRef ] [ Medline ]
• Thomas J, Prabhu AV, Heron DE, Beriwal S. Twitter and brachytherapy: An analysis of "tweets" over six years by patients and health care professionals. Brachytherapy. 2018;17(6):1004-1010. [ CrossRef ] [ Medline ]
• Kelly-Hedrick M, Grunberg PH, Brochu F, Zelkowitz P. "It's totally okay to be sad, but never lose hope": content analysis of infertility-related videos on YouTube in relation to viewer preferences. J Med Internet Res. May 23, 2018;20(5):e10199. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gage-Bouchard EA, LaValley S, Mollica M, Beaupin LK. Cancer communication on social media: examining how cancer caregivers use Facebook for cancer-related communication. Cancer Nurs. 2017;40(4):332-338. [ CrossRef ] [ Medline ]
• Rael CT, Pierre D, Frye V, Kessler D, Duffy L, Malos N, et al. Evaluating blood donor experiences and barriers/facilitators to blood donation in the United States using YouTube video content. Transfusion. Sep 2021;61(9):2650-2657. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fisher S, Jehassi A, Ziv M. Hidradenitis suppurativa on Facebook: thematic and content analyses of patient support group. Arch Dermatol Res. Aug 2020;312(6):421-426. [ CrossRef ] [ Medline ]
• Abdoli S, Hessler D, Vora A, Smither B, Stuckey H. Descriptions of diabetes burnout from individuals with Type 1 diabetes: an analysis of YouTube videos. Diabet Med. Aug 2020;37(8):1344-1351. [ CrossRef ] [ Medline ]
• Myneni S, Lewis B, Singh T, Paiva K, Kim SM, Cebula AV, et al. Diabetes self-management in the age of social media: large-scale analysis of peer interactions using semiautomated methods. JMIR Med Inform. Jun 30, 2020;8(6):e18441. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Charlie AM, Gao Y, Heller SL. What do patients want to know? questions and concerns regarding mammography expressed through social media. J Am Coll Radiol. Oct 2018;15(10):1478-1486. [ CrossRef ] [ Medline ]
• Watts G, Christou P, Antonarakis G. Experiences of individuals concerning combined orthodontic and orthognathic surgical treatment: a qualitative Twitter analysis. Med Princ Pract. 2018;27(3):227-235. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pai RR, Alathur S. Assessing mobile health applications with Twitter analytics. Int J Med Inform. May 2018;113:72-84. [ CrossRef ] [ Medline ]
• Cheng TY, Liu L, Woo BK. Analyzing Twitter as a platform for Alzheimer-related dementia awareness: thematic analyses of tweets. JMIR Aging. Dec 10, 2018;1(2):e11542. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bridges N, Howell G, Schmied V. Exploring breastfeeding support on social media. Int Breastfeed J. 2018;13:22. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Anderson JG, Hundt E, Dean M, Keim-Malpass J, Lopez RP. "The Church of Online Support": examining the use of blogs among family caregivers of persons with dementia. J Fam Nurs. Feb 2017;23(1):34-54. [ CrossRef ] [ Medline ]
• Kearney MD, Selvan P, Hauer MK, Leader AE, Massey PM. Characterizing HPV vaccine sentiments and content on Instagram. Health Educ Behav. Dec 2019;46(2_suppl):37-48. [ CrossRef ] [ Medline ]
• Davies SH, Langer MD, Klein A, Gonzalez-Hernandez G, Dowshen N. Adolescent perceptions of menstruation on Twitter: opportunities for advocacy and education. J Adolesc Health. Jul 2022;71(1):94-104. [ CrossRef ] [ Medline ]
• Van Diepen C, Rosales Valdes D. A content analysis on the perceptions of LGBTQ+ (centred) health care on Twitter. Health Expect. Dec 2022;25(6):3238-3245. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pleasure ZH, Frohwirth LF, Li N, Polis CB. A content analysis of Reddit users' posts about challenges to contraceptive care-seeking during COVID-19-related restrictions in the United States. J Health Commun. Oct 03, 2022;27(10):746-754. [ CrossRef ] [ Medline ]
• Du Y, Dennis B, Ramirez V, Li C, Wang J, Meireles CL. Experiences and disease self-management in individuals living with chronic kidney disease: qualitative analysis of the National Kidney Foundation's online community. BMC Nephrol. Mar 04, 2022;23(1):88. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sadek Attalla S, Ow NL, McNarry M, De Simoni A. Experiences of exercise in patients with asthma: a qualitative analysis of discussions in a UK asthma online community. BJGP Open. Apr 29, 2022;6(3):BJGPO.2021.0162. [ CrossRef ]
• Bartmess M, Talbot C, O'Dwyer ST, Lopez RP, Rose KM, Anderson JG. Using Twitter to understand perspectives and experiences of dementia and caregiving at the beginning of the COVID-19 pandemic. Dementia (London). Jul 2022;21(5):1734-1752. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brewer G, Centifanti L, Caicedo JC, Huxley G, Peddie C, Stratton K, et al. Experiences of mental distress during COVID-19: thematic analysis of discussion forum posts for anxiety, depression, and obsessive-compulsive disorder. Illn Crises Loss. Oct 2022;30(4):795-811. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Castillo LIR, Hadjistavropoulos T, Beahm J. Social media discussions about long-term care and the COVID-19 pandemic. J Aging Stud. Dec 2022;63:101076. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Colaceci S, Anderson G, Ricciuto V, Montinaro D, Alazraki G, Mena-Tudela D. Experiences of birth during COVID-19 pandemic in Italy and Spain: A thematic analysis. Int J Environ Res Public Health. Jun 18, 2022;19(12):7488. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tripathi SD, Parker PD, Prabhu AV, Thomas K, Rodriguez A. An examination of patients and caregivers on Reddit Navigating Brain Cancer: content analysis of the brain tumor Subreddit. JMIR Cancer. Jun 22, 2022;8(2):e35324. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jina-Pettersen N. Fear, neglect, coercion, and dehumanization: is inpatient psychiatric trauma contributing to a public health crisis? J Patient Exp. 2022;9:23743735221079138. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rossi NA, Devarajan K, Chokshi SN, Ochoa VJ, Benavidez M, Malaya LT, et al. Social media depictions of cochlear implants: An Instagram and TikTok analysis. Otol Neurotol. 2023;44(1):e13-e21. [ CrossRef ]
• Singh GK, Rego J, Chambers S, Fox J. Health professionals' perspectives of the role of palliative care during COVID-19: content analysis of articles and blogs posted on Twitter. Am J Hosp Palliat Care. Apr 2022;39(4):487-493. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Koly KN, Tasnim Z, Ahmed S, Saba J, Mahmood R, Farin FT, et al. Mental healthcare-seeking behavior of women in Bangladesh: content analysis of a social media platform. BMC Psychiatry. Dec 19, 2022;22(1):797. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Belcher R, Sim D, Meykler M, Owens-Walton J, Hassan N, Rubin R, et al. A qualitative analysis of female Reddit users' experiences with low libido: how do women perceive their changes in sexual desire? J Sex Med. Feb 27, 2023;20(3):287-297. [ CrossRef ] [ Medline ]
• Culp F, Wu Y, Wu D, Ren Y, Raynor P, Hung P, et al. Understanding alcohol use discourse and stigma patterns in perinatal care on Twitter. Healthcare (Basel). Nov 26, 2022;10(12):2375. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ferrey A, Ashworth G, Cabling M, Rundblad G, Ismail K. A thematic analysis of YouTube comments on a television documentary titled 'Diabulimia: The world's most dangerous eating disorder'. Diabet Med. May 2023;40(5):e15025. [ CrossRef ] [ Medline ]
• Watt S, Salway T, Gómez-Ramírez O, Ablona A, Barton L, Chang H, et al. Rumination, risk, and response: a qualitative analysis of sexual health anxiety among online sexual health chat service users. Sex. Health. May 23, 2022;19(3):182-191. [ CrossRef ]
• Naganathan G, Bilgen I, Cleland J, Reel E, Cil T. #COVID19 and #Breastcancer: a qualitative analysis of tweets. Curr Oncol. Nov 08, 2022;29(11):8483-8500. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Potgieter I, Hoare DJ, Fackrell K. Hyperacusis in children: a thematic analysis of discussions in online forums. Am J Audiol. Mar 03, 2022;31(1):166-174. [ CrossRef ]
• Manning Hutson M, Hosking SM, Mantalvanos S, Berk M, Pasco J, Dunning T. What injured workers with complex claims look for in online communities: netnographic analysis. J Med Internet Res. Apr 07, 2022;24(4):e17180. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lawless MT, Hunter SC, Pinero de Plaza MA, Archibald MM, Kitson AL. "You Are By No Means Alone": a netnographic study of self-care support in an online community for older adults. Qual Health Res. Nov 2022;32(13):1935-1951. [ CrossRef ] [ Medline ]
• Keim-Malpass J, Mitchell EM, Sun E, Kennedy C. Using Twitter to understand public perceptions regarding the #HPV vaccine: opportunities for public health nurses to engage in social marketing. Public Health Nurs. Jul 2017;34(4):316-323. [ CrossRef ] [ Medline ]
• Yamada R, Rasmussen KM, Felice JP. "What is 'enough,' and how do I make it?": a qualitative examination of questions mothers ask on social media about pumping and providing an adequate amount of milk for their infants. Breastfeed Med. 2019;14(1):17-21. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Osadchiy V, Mills JN, Eleswarapu SV. Understanding patient anxieties in the social media era: qualitative analysis and natural language processing of an online male infertility community. J Med Internet Res. Mar 10, 2020;22(3):e16728. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Thomas TH, Nauth-Shelley K, Thompson MA, Attai DJ, Katz MS, Graham D, et al. The needs of women treated for ovarian cancer: results from a #gyncsm Twitter chat. J Patient Cent Res Rev. 2018;5(2):149-157. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mehta N, Zhu L, Lam K, Stall NM, Savage R, Read SH, et al. Health forums and Twitter for dementia research: opportunities and considerations. J Am Geriatr Soc. Dec 2020;68(12):2881-2889. [ CrossRef ] [ Medline ]
• Gupta R, Ariefdjohan M. Mental illness on Instagram: a mixed method study to characterize public content, sentiments, and trends of antidepressant use. J Ment Health. Aug 2021;30(4):518-525. [ CrossRef ] [ Medline ]
• He L, He C, Reynolds T, Bai Q, Huang Y, Li C, et al. Why do people oppose mask wearing? A comprehensive analysis of U.S. tweets during the COVID-19 pandemic. J Am Med Inform Assoc. Jul 14, 2021;28(7):1564-1573. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Loeb S, Mihalcea R, Perez-Rosas V, Xu A, Taylor J, Byrne N, et al. Leveraging social media as a thermometer to gauge patient and caregiver concerns: COVID-19 and prostate cancer. Eur Urol Open Sci. Mar 2021;25:1-4. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Graf I, Gerwing H, Hoefer K, Ehlebracht D, Christ H, Braumann B. Social media and orthodontics: A mixed-methods analysis of orthodontic-related posts on Twitter and Instagram. Am J Orthod Dentofacial Orthop. Aug 2020;158(2):221-228. [ CrossRef ] [ Medline ]
• Boon-Itt S, Skunkan Y. Public perception of the COVID-19 pandemic on Twitter: sentiment analysis and topic modeling study. JMIR Public Health Surveill. Nov 11, 2020;6(4):e21978. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wang J, Zhou Y, Zhang W, Evans R, Zhu C. Concerns expressed by Chinese social media users during the COVID-19 pandemic: content analysis of Sina Weibo microblogging data. J Med Internet Res. Nov 26, 2020;22(11):e22152. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wahbeh A, Nasralah T, Al-Ramahi M, El-Gayar O. Mining physicians' opinions on social media to obtain insights into COVID-19: mixed methods analysis. JMIR Public Health Surveill. Jun 18, 2020;6(2):e19276. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Çınar S, Boztepe H, Prof. The use of social media among parents of infants with cleft lip and/or palate. J Pediatr Nurs. 2020;54:e91-e96. [ CrossRef ] [ Medline ]
• Jiang T, Osadchiy V, Mills JN, Eleswarapu SV. Is It all in my head? self-reported psychogenic erectile dysfunction and depression are common among young men seeking advice on social media. Urology. Aug 2020;142:133-140. [ CrossRef ] [ Medline ]
• Dzubur E, Khalil C, Almario CV, Noah B, Minhas D, Ishimori M, et al. Patient concerns and perceptions regarding biologic therapies in ankylosing spondylitis: insights from a large-scale survey of social media platforms. Arthritis Care Res (Hoboken). Feb 2019;71(2):323-330. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gonzalez G, Vaculik K, Khalil C, Zektser Y, Arnold C, Almario CV, et al. Women's experience with stress urinary incontinence: insights from social media analytics. Journal of Urology. May 2020;203(5):962-968. [ CrossRef ]
• Jha SR, McDonagh J, Prichard R, Newton PJ, Hickman LD, Fung E, et al. #Frailty: A snapshot Twitter report on frailty knowledge translation. Australas J Ageing. Dec 2018;37(4):309-312. [ CrossRef ] [ Medline ]
• Litchman ML, Snider C, Edelman LS, Wawrzynski SE, Gee PM. Diabetes online community user perceptions of successful aging with diabetes: analysis of a #DSMA Tweet chat. JMIR Aging. Jun 22, 2018;1(1):e10176. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jones J, Pradhan M, Hosseini M, Kulanthaivel A, Hosseini M. Novel approach to cluster patient-generated data into actionable topics: case study of a web-based breast cancer forum. JMIR Med Inform. Nov 29, 2018;6(4):e45. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gonzalez G, Vaculik K, Khalil C, Zektser Y, Arnold C, Almario CV, et al. Using large-scale social media analytics to understand patient perspectives about urinary tract infections: thematic analysis. J Med Internet Res. Jan 25, 2022;24(1):e26781. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sormunen T, Westerbotn M, Aanesen A, Fossum B, Karlgren K. Social media in the infertile community-using a text analysis tool to identify the topics of discussion on the multitude of infertility blogs. Womens Health (Lond). 2021;17:17455065211063280. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Berkovic D, Ackerman IN, Briggs AM, Ayton D. Tweets by people with arthritis during the COVID-19 pandemic: content and sentiment analysis. J Med Internet Res. Dec 03, 2020;22(12):e24550. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Della Rosa S, Sen F. Health topics on Facebook Groups: content analysis of posts in multiple sclerosis communities. Interact J Med Res. Feb 11, 2019;8(1):e10146. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Awofeso N, Imam SA, Ahmed A. Content analysis of media coverage of childhood obesity topics in UAE newspapers and popular social media platforms, 2014-2017. Int J Health Policy Manag. Feb 01, 2019;8(2):81-89. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Smith DJ, Mac VV, Hertzberg VS. Using Twitter for nursing research: a tweet analysis on heat illness and health. J Nurs Scholarsh. May 2021;53(3):343-350. [ CrossRef ] [ Medline ]
• Green BM, Van Horn KT, Gupte K, Evans M, Hayes S, Bhowmick A. Assessment of adaptive engagement and support model for people with chronic health conditions in online health communities: combined content analysis. J Med Internet Res. Jul 07, 2020;22(7):e17338. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Osakwe ZT, Cortés YI. Impact of COVID-19: a text mining analysis of Twitter data in Spanish language. Hisp Health Care Int. Dec 2021;19(4):239-245. [ CrossRef ] [ Medline ]
• Sümeyye Yorulmaz D, Karadeniz H. Vaccination refusal debate on social media in Turkey: a content analysis of the comments on Instagram blogs. Iran J Public Health. Mar 2022;51(3):615-623. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Choi E, Becker H, Kim S. A blog text analysis to explore psychosocial support in adolescents and young adults with cancer. Cancer Nurs. Dec 11, 2022;46(2):143-151. [ CrossRef ]
• Jun J, Wickersham K, Zain A, Ford R, Zhang N, Ciccarelli C, et al. Cancer and COVID-19 vaccines on Twitter: the voice and vaccine attitude of cancer community. J Health Commun. Jan 02, 2023;28(1):1-14. [ CrossRef ] [ Medline ]
• Yashpal S, Raghunath A, Gencerliler N, Burns LE. Exploring public perceptions of dental care affordability in the United States: mixed method analysis via Twitter. JMIR Form Res. Jul 01, 2022;6(7):e36315. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Damier P, Henderson EJ, Romero-Imbroda J, Galimam L, Kronfeld N, Warnecke T. Impact of off-time on quality of life in Parkinson's patients and their caregivers: insights from social media. Parkinsons Dis. 2022;2022:1800567. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Miller WR, Malloy C, Mravec M, Sposato MF, Groves D. Nursing in the spotlight: Talk about nurses and the nursing profession on Twitter during the early COVID-19 pandemic. Nurs Outlook. 2022;70(4):580-589. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hriberšek M, Eibensteiner F, Kapral L, Teufel A, Nawaz FA, Cenanovic M, et al. "Loved ones are not 'visitors' in a patient's life"-The importance of including loved ones in the patient's hospital stay: An international Twitter study of #HospitalsTalkToLovedOnes in times of COVID-19. Front Public Health. 2023;11:1100280. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Sullivan L, Killeen RP, Doran P, Crowley RK. Adherence with reporting of ethical standards in COVID-19 human studies: a rapid review. BMC Med Ethics. Jun 28, 2021;22(1):80. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Denecke K, Bamidis P, Bond C, Gabarron E, Househ M, Lau AYS, et al. Ethical issues of social media usage in healthcare. Yearb Med Inform. Aug 13, 2015;10(1):137-147. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ferretti A, Ienca M, Sheehan M, Blasimme A, Dove ES, Farsides B, et al. Ethics review of big data research: What should stay and what should be reformed? BMC Med Ethics. Apr 30, 2021;22(1):51. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kohn T, Shore C. The Ethics of University Ethics Committees: risk management and the research imagination. Death Public Univ Uncertain Futur High Educ Knowl Econ. May 2017:229-249. [ CrossRef ]
• Friesen P, Redman B, Caplan A. Of straws, camels, research regulation, and IRBs. Ther Innov Regul Sci. Jul 2019;53(4):526-534. [ CrossRef ] [ Medline ]
• Sellers C, Samuel G, Derrick G. Reasoning "uncharted territory": notions of expertise within ethics review panels assessing research use of social media. J Empir Res Hum Res Ethics. 2020;15(1-2):28-39. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kious B. The Nuremberg Code: its history and implications. Princet J Bioeth. 2001;4:7-19. [ Medline ]
• World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. Nov 27, 2013;310(20):2191-2194. [ CrossRef ] [ Medline ]
• Department of Health‚ Education‚Welfare, National Commission for the Protection of Human Subjects of BiomedicalBehavioral Research. The Belmont Report. Ethical principles and guidelines for the protection of human subjects of research. J Am Coll Dent. 2014;81(3):4-13. [ Medline ]
• Wongkoblap A, Vadillo MA, Curcin V. Researching mental health disorders in the era of social media: systematic review. J Med Internet Res. Jun 29, 2017;19(6):e228. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alim S. An initial exploration of ethical research practices regarding automated data extraction from online social media user profiles. First Monday. Jul 2014;19(7):105-127. [ CrossRef ] [ Medline ]
• Vitak J, Proferes N, Shilton K, Ashktorab Z. Ethics regulation in social computing research: examining the role of institutional review boards. J Empir Res Hum Res Ethics. Dec 2017;12(5):372-382. [ CrossRef ] [ Medline ]
• Moreno MA, Goniu N, Moreno PS, Diekema D. Ethics of social media research: common concerns and practical considerations. Cyberpsychol Behav Soc Netw. Sep 2013;16(9):708-713. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Metcalf J, Crawford K. Where are human subjects in Big Data research? The emerging ethics divide. Big Data & Society. Jun 01, 2016;3(1):205395171665021. [ CrossRef ]
• Usha Lawrance J, Nayahi Jesudhasan JV. Privacy preserving parallel clustering based anonymization for big data using MapReduce framework. Applied Artificial Intelligence. Oct 17, 2021;35(15):1587-1620. [ CrossRef ]
• Wilkinson D, Thelwall M. Researching personal information on the public web. Social Science Computer Review. Aug 17, 2010;29(4):387-401. [ CrossRef ]
• International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals: writing and editing for biomedical publication. Croat Med J. Dec 2003;44(6):770-783. [ FREE Full text ] [ Medline ]
• Boyd D, Crawford K. Critical questions for big data. Information, Communication & Society. Jun 2012;15(5):662-679. [ CrossRef ]
• Bond CS, Ahmed OH, Hind M, Thomas B, Hewitt-Taylor J. The conceptual and practical ethical dilemmas of using health discussion board posts as research data. J Med Internet Res. Jun 07, 2013;15(6):e112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Markham A, Tiidenberg K, Herman A. Ethics as methods: doing ethics in the era of big data research—introduction. Social Media + Society. Jul 19, 2018;4(3):205630511878450-205630511878416. [ CrossRef ]

Abbreviations

Edited by T Leung, T de Azevedo Cardoso; submitted 02.08.23; peer-reviewed by E Zibrowski, J Scheibner; comments to author 06.10.23; revised version received 29.11.23; accepted 16.04.24; published 17.05.24.

©Yujie Zhang, Jiaqi Fu, Jie Lai, Shisi Deng, Zihan Guo, Chuhan Zhong, Jianyao Tang, Wenqiong Cao, Yanni Wu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• Open access
• Published: 13 May 2024

Sexual and reproductive health implementation research in humanitarian contexts: a scoping review

• Alexandra Norton 1 &
• Hannah Tappis 2  

Reproductive Health volume  21 , Article number:  64 ( 2024 ) Cite this article

85 Accesses

Metrics details

Meeting the health needs of crisis-affected populations is a growing challenge, with 339 million people globally in need of humanitarian assistance in 2023. Given one in four people living in humanitarian contexts are women and girls of reproductive age, sexual and reproductive health care is considered as essential health service and minimum standard for humanitarian response. Despite growing calls for increased investment in implementation research in humanitarian settings, guidance on appropriate methods and analytical frameworks is limited.

A scoping review was conducted to examine the extent to which implementation research frameworks have been used to evaluate sexual and reproductive health interventions in humanitarian settings. Peer-reviewed papers published from 2013 to 2022 were identified through relevant systematic reviews and a literature search of Pubmed, Embase, PsycInfo, CINAHL and Global Health databases. Papers that presented primary quantitative or qualitative data pertaining to a sexual and reproductive health intervention in a humanitarian setting were included.

Seven thousand thirty-six unique records were screened for inclusion, and 69 papers met inclusion criteria. Of these, six papers explicitly described the use of an implementation research framework, three citing use of the Consolidated Framework for Implementation Research. Three additional papers referenced other types of frameworks used in their evaluation. Factors cited across all included studies as helping the intervention in their presence or hindering in their absence were synthesized into the following Consolidated Framework for Implementation Research domains: Characteristics of Systems, Outer Setting, Inner Setting, Characteristics of Individuals, Intervention Characteristics, and Process.

This review found a wide range of methodologies and only six of 69 studies using an implementation research framework, highlighting an opportunity for standardization to better inform the evidence for and delivery of sexual and reproductive health interventions in humanitarian settings. Increased use of implementation research frameworks such as a modified Consolidated Framework for Implementation Research could work toward both expanding the evidence base and increasing standardization.

Plain English summary

Three hundred thirty-nine million people globally were in need of humanitarian assistance in 2023, and meeting the health needs of crisis-affected populations is a growing challenge. One in four people living in humanitarian contexts are women and girls of reproductive age, and provision of sexual and reproductive health care is considered to be essential within a humanitarian response. Implementation research can help to better understand how real-world contexts affect health improvement efforts. Despite growing calls for increased investment in implementation research in humanitarian settings, guidance on how best to do so is limited. This scoping review was conducted to examine the extent to which implementation research frameworks have been used to evaluate sexual and reproductive health interventions in humanitarian settings. Of 69 papers that met inclusion criteria for the review, six of them explicitly described the use of an implementation research framework. Three used the Consolidated Framework for Implementation Research, a theory-based framework that can guide implementation research. Three additional papers referenced other types of frameworks used in their evaluation. This review summarizes how factors relevant to different aspects of implementation within the included papers could have been organized using the Consolidated Framework for Implementation Research. The findings from this review highlight an opportunity for standardization to better inform the evidence for and delivery of sexual and reproductive health interventions in humanitarian settings. Increased use of implementation research frameworks such as a modified Consolidated Framework for Implementation Research could work toward both expanding the evidence base and increasing standardization.

Peer Review reports

Over the past few decades, the field of public health implementation research (IR) has grown as a means by which the real-world conditions affecting health improvement efforts can be better understood. Peters et al. put forward the following broad definition of IR for health: “IR is the scientific inquiry into questions concerning implementation – the act of carrying an intention into effect, which in health research can be policies, programmes, or individual practices (collectively called interventions)” [ 1 ].

As IR emphasizes real-world circumstances, the context within which a health intervention is delivered is a core consideration. However, much IR implemented to date has focused on higher-resource settings, with many proposed frameworks developed with particular utility for a higher-income setting [ 2 ]. In recognition of IR’s potential to increase evidence across a range of settings, there have been numerous reviews of the use of IR in lower-resource settings as well as calls for broader use [ 3 , 4 ]. There have also been more focused efforts to modify various approaches and frameworks to strengthen the relevance of IR to low- and middle-income country settings (LMICs), such as the work by Means et al. to adapt a specific IR framework for increased utility in LMICs [ 2 ].

Within LMIC settings, the centrality of context to a health intervention’s impact is of particular relevance in humanitarian settings, which present a set of distinct implementation challenges [ 5 ]. Humanitarian responses to crisis situations operate with limited resources, under potential security concerns, and often under pressure to relieve acute suffering and need [ 6 ]. Given these factors, successful implementation of a particular health intervention may require different qualities than those that optimize intervention impact under more stable circumstances [ 7 ]. Despite increasing recognition of the need for expanded evidence of health interventions in humanitarian settings, the evidence base remains limited [ 8 ]. Furthermore, despite its potential utility, there is not standardized guidance on IR in humanitarian settings, nor are there widely endorsed recommendations for the frameworks best suited to analyze implementation in these settings.

Sexual and reproductive health (SRH) is a core aspect of the health sector response in humanitarian settings [ 9 ]. Yet, progress in addressing SRH needs has lagged far behind other services because of challenges related to culture and ideology, financing constraints, lack of data and competing priorities [ 10 ]. The Minimum Initial Service Package (MISP) for SRH in Crisis Situations is the international standard for the minimum set of SRH services that should be implemented in all crisis situations [ 11 ]. However, as in other areas of health, there is need for expanded evidence for planning and implementation of SRH interventions in humanitarian settings. Recent systematic reviews of SRH in humanitarian settings have focused on the effectiveness of interventions and service delivery strategies, as well as factors affecting utilization, but have not detailed whether IR frameworks were used [ 12 , 13 , 14 , 15 ]. There have also been recent reviews examining IR frameworks used in various settings and research areas, but none have explicitly focused on humanitarian settings [ 2 , 16 ].

Given the need for an expanded evidence base for SRH interventions in humanitarian settings and the potential for IR to be used to expand the available evidence, a scoping review was undertaken. This scoping review sought to identify IR approaches that have been used in the last ten years to evaluate SRH interventions in humanitarian settings.

This review also sought to shed light on whether there is a need for a common framework to guide research design, analysis, and reporting for SRH interventions in humanitarian settings and if so, if there are any established frameworks already in use that would be fit-for-purpose or could be tailored to meet this need.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping reviews was utilized to guide the elements of this review [ 17 ]. The review protocol was retrospectively registered with the Open Science Framework ( https://osf.io/b5qtz ).

Search strategy

A two-fold search strategy was undertaken for this review, which covered the last 10 years (2013–2022). First, recent systematic reviews pertaining to research or evaluation of SRH interventions in humanitarian settings were identified through keyword searches on PubMed and Google Scholar. Four relevant systematic reviews were identified [ 12 , 13 , 14 , 15 ] Table 1 .

Second, a literature search mirroring these reviews was conducted to identify relevant papers published since the completion of searches for the most recent review (April 2017). Additional file 1 includes the search terms that were used in the literature search [see Additional file 1 ].

The literature search was conducted for papers published from April 2017 to December 2022 in the databases that were searched in one or more of the systematic reviews: PubMed, Embase, PsycInfo, CINAHL and Global Health. Searches were completed in January 2023 Table 2 .

Two reviewers screened each identified study for alignment with inclusion criteria. Studies in the four systematic reviews identified were considered potentially eligible if published during the last 10 years. These papers then underwent full-text review to confirm satisfaction of all inclusion criteria, as inclusion criteria were similar but not fully aligned across the four reviews.

Literature search results were exported into a citation manager (Covidence), duplicates were removed, and a step-wise screening process for inclusion was applied. First, all papers underwent title and abstract screening. The remaining papers after abstract screening then underwent full-text review to confirm satisfaction of all inclusion criteria. Title and abstract screening as well as full-text review was conducted independently by both authors; disagreements after full-text review were resolved by consensus.

Data extraction and synthesis

The following content areas were summarized in Microsoft Excel for each paper that met inclusion criteria: publication details including author, year, country, setting [rural, urban, camp, settlement], population [refugees, internally displaced persons, general crisis-affected], crisis type [armed conflict, natural disaster], crisis stage [acute, chronic], study design, research methods, SRH intervention, and intervention target population [specific beneficiaries of the intervention within the broader population]; the use of an IR framework; details regarding the IR framework, how it was used, and any rationale given for the framework used; factors cited as impacting SRH interventions, either positively or negatively; and other key findings deemed relevant to this review.

As the focus of this review was on the approach taken for SRH intervention research and evaluation, the quality of the studies themselves was not assessed.

Twenty papers underwent full-text review due to their inclusion in one or more of the four systematic reviews and meeting publication date inclusion criteria. The literature search identified 7,016 unique papers. After full-text screening, 69 met all inclusion criteria and were included in the review. Figure  1 illustrates the search strategy and screening process.

Flow chart of paper identification

Papers published in each of the 10 years of the review timeframe (2013–2022) were included. 29% of the papers originated from the first five years of the time frame considered for this review, with the remaining 71% papers coming from the second half. Characteristics of included publications, including geographic location, type of humanitarian crisis, and type of SRH intervention, are presented in Table  3 .

A wide range of study designs and methods were used across the papers, with both qualitative and quantitative studies well represented. Twenty-six papers were quantitative evaluations [ 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ], 17 were qualitative [ 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ], and 26 used mixed methods [ 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 ]. Within the quantitative evaluations, 15 were observational, while five were quasi-experimental, five were randomized controlled trials, and one was an economic evaluation. Study designs as classified by the authors of this review are summarized in Table  4 .

Six papers (9%) explicitly cited use of an IR framework. Three of these papers utilized the Consolidated Framework for Implementation Research (CFIR) [ 51 , 65 , 70 ]. The CFIR is a commonly used determinant framework that—in its originally proposed form in 2009—is comprised of five domains, each of which has constructs to further categorize factors that impact implementation. The CFIR domains were identified as core content areas influencing the effectiveness of implementation, and the constructs within each domain are intended to provide a range of options for researchers to select from to “guide diagnostic assessments of implementation context, evaluate implementation progress, and help explain findings.” [ 87 ] To allow for consistent terminology throughout this review, the original 2009 CFIR domains and constructs are used.

Guan et al. conducted a mixed methods study to assess the feasibility and effectiveness of a neonatal hepatitis B immunization program in a conflict-affected rural region of Myanmar. Guan et al. report mapping data onto the CFIR as a secondary analysis step. They describe that “CFIR was used as a comprehensive meta-theoretical framework to examine the implementation of the Hepatitis B Virus vaccination program,” and implementation themes from multiple study data sources (interviews, observations, examination of monitoring materials) were mapped onto CFIR constructs. They report their results in two phases – Pre-implementation training and community education, and Implementation – with both anchored in themes that they had mapped onto CFIR domains and constructs. All but six constructs were included in their analysis, with a majority summarized in a table and key themes explored further in the narrative text. They specify that most concerns were identified within the Outer Setting and Process domains, while elements identified within the Inner Setting domain provided strength to the intervention and helped mitigate against barriers [ 70 ].

Sarker et al. conducted a qualitative study to assess provision of maternal, newborn and child health services to Rohingya refugees residing in camps in Cox’s Bazar, Bangladesh. They cite using CFIR as a guide for thematic analysis, applying it after a process of inductive and deductive coding to index these codes into the CFIR domains. They utilized three of the five CFIR domains (Outer Setting, Inner Setting, and Process), stating that the remaining two domains (Intervention Characteristics and Characteristics of Individuals) were not relevant to their analysis. They then proposed two additional CFIR domains, Context and Security, for use in humanitarian contexts. In contrast to Guan et al., CFIR constructs are not used nor mentioned by Sarker et al., with content under each domain instead synthesized as challenges and potential solutions. Regarding the CFIR, Sarker et al. write, “The CFIR guided us for interpretative coding and creating the challenges and possible solutions into groups for further clarification of the issues related to program delivery in a humanitarian crisis setting.” [ 51 ]

Sami et al. conducted a mixed methods case study to assess the implementation of a package of neonatal interventions at health facilities within refugee and internally displaced persons camps in South Sudan. They reference use of the CFIR earlier in the study than Sarker et al., basing their guides for semi-structured focus group discussions on the CFIR framework. They similarly reference a general use of the CFIR framework as they conducted thematic analysis. Constructs are referenced once, but they do not specify whether their application of the CFIR framework included use of domains, constructs, or both. This may be in part because they then applied an additional framework, the World Health Organization (WHO) Health System Framework, to present their findings. They describe a nested approach to their use of these frameworks: “Exploring these [CFIR] constructs within the WHO Health Systems Framework can identify specific entry points to improve the implementation of newborn interventions at critical health system building blocks.” [ 65 ]

Three papers cite use of different IR frameworks. Bolan et al. utilized the Theoretical Domains Framework in their mixed methods feasibility study and pilot cluster randomized trial evaluating pilot use of the Safe Delivery App by maternal and newborn health workers providing basic emergency obstetric and newborn care in facilities in the conflict-affected Maniema province of the Democratic Republic of the Congo (DRC). They used the Theroetical Domains Framework in designing interview questions, and further used it as the coding framework for their analysis. Similar to the CFIR, the Theoretical Domains Framework is a determinant framework that consists of domains, each of which then includes constructs. Bolan et al. utilized the Theoretical Domains Framework at the construct level in interview question development and at the domain level in their analysis, mapping interview responses to eight of the 14 domains [ 83 ]. Berg et al. report using an “exploratory design guided by the principles of an evaluation framework” developed by the Medical Research Council to analyze the implementation process, mechanisms of impact, and outcomes of a three-pillar training intervention to improve maternal and neonatal healthcare in the conflict-affected South Kivu province of the DRC [ 67 , 88 ]. Select components of this evaluation framework were used to guide deductive analysis of focus group discussions and in-depth interviews [ 67 ]. In their study of health workers’ knowledge and attitudes toward newborn health interventions in South Sudan, before and after training and supply provision, Sami et al. report use of the Conceptual Framework of the Role of Attitudes in Evidence-Based Practice Implementation in their analysis process. The framework was used to group codes following initial inductive coding analysis of in-depth interviews [ 72 ].

Three other papers cite use of specific frameworks in their intervention evaluation [ 19 , 44 , 76 ]. As a characteristic of IR is the use of an explicit framework to guide the research, the use of the frameworks in these three papers meets the intention of IR and serves the purpose that an IR framework would have in strengthening the analytical rigor. Castle et al. cite use of their program’s theory of change as a framework for a mixed methods evaluation of the provision of family planning services and more specifically uptake of long-acting reversible contraception use in the DRC. They describe use of the theory of change to “enhance effectiveness of [long-acting reversible contraception] access and uptake.” [ 76 ] Thommesen et al. cite use of the AAAQ (Availability, Accessibility, Acceptability and Quality) framework in their qualitative study assessing midwifery services provided to pregnant women in Afghanistan. This framework is focused on the “underlying elements needed for attainment of optimum standard of health care,” but the authors used it in this paper to evaluate facilitators and barriers to women accessing midwifery services [ 44 ]. Jarrett et al. cite use of the Centers for Disease Control and Prevention’s (CDC) Guidelines for Evaluating Public Health Surveillance Systems to explore the characteristics of a population mobility, mortality and birth surveillance system in South Kivu, DRC. Use of these CDC guidelines is cited as one of four study objectives, and commentary is included in the Results section pertaining to each criteria within these guidelines, although more detail regarding use of these guidelines or the authors’ experience with their use in the study is not provided [ 19 ].

Overall, 22 of the 69 papers either explicitly or implicitly identified IR as relevant to their work. Nineteen papers include a focus on feasibility (seven of which did not otherwise identify the importance of exploring questions concerning implementation), touching on a common outcome of interest in implementation research [ 89 ].

While a majority of papers did not explicitly or implicitly use an IR framework to evaluate their SRH intervention of focus, most identified factors that facilitated implementation when they were present or served as a barrier when absent. Sixty cite factors that served as facilitators and 49 cite factors that served as barriers, with just three not citing either. Fifty-nine distinct factors were identified across the papers.

Three of the six studies that explicitly used an IR framework used the CFIR, and the CFIR is the only IR framework that was used by multiple studies. As previously mentioned, Means et al. put forth an adaptation of the CFIR to increase its relevance in LMIC settings, proposing a sixth domain (Characteristics of Systems) and 11 additional constructs [ 2 ]. Using the expanded domains and constructs as proposed by Means et al., the 59 factors cited by papers in this review were thematically grouped into the six domains: Characteristics of Systems, Outer Setting, Inner Setting, Characteristics of Individuals, Intervention Characteristics, and Process. Within each domain, alignment with CFIR constructs was assessed for, and alignment was found with 29 constructs: eight of Means et al.’s 11 constructs, and 21 of the 39 standard CFIR constructs. Three factors did not align with any construct (all fitting within the Outer Setting domain), and 14 aligned with a construct label but not the associated definition. Table 5 synthesizes the mapping of factors affecting SRH intervention implementation to CFIR domains and constructs, with the construct appearing in italics if it is considered to align with that factor by label but not by definition.

Table 6 lists the CFIR constructs that were not found to have alignment with any factor cited by the papers in this review.

This scoping review sought to assess how IR frameworks have been used to bolster the evidence base for SRH interventions in humanitarian settings, and it revealed that IR frameworks, or an explicit IR approach, are rarely used. All four of the systematic reviews identified with a focus on SRH in humanitarian settings articulate the need for more research examining the effectiveness of SRH interventions in humanitarian settings, with two specifically citing a need for implementation research/science [ 12 , 13 ]. The distribution of papers across the timeframe included in this review does suggest that more research on SRH interventions for crisis-affected populations is taking place, as a majority of relevant papers were published in the second half of the review period. The papers included a wide range of methodologies, which reflect the differing research questions and contexts being evaluated. However, it also invites the question of whether there should be more standardization of outcomes measured or frameworks used to guide analysis and to facilitate increased comparison, synthesis and application across settings.

Three of the six papers that used an IR framework utilized the CFIR. Guan et al. used the CFIR at both a domain and construct level, Sarker et al. used the CFIR at the domain level, and Sami et al. did not specify which CFIR elements were used in informing the focus group discussion guide [ 51 , 65 , 70 ]. It is challenging to draw strong conclusions about the applicability of CFIR in humanitarian settings based on the minimal use of CFIR and IR frameworks within the papers reviewed, although Guan et al. provides a helpful model for how analysis can be structured around CFIR domains and constructs. It is worth considering that the minimal use of IR frameworks, and more specifically CFIR constructs, could be in part because that level of prescriptive categorization does not allow for enough fluidity in humanitarian settings. It also raises questions about the appropriate degree of standardization to pursue for research done in these settings.

The mapping of factors affecting SRH intervention implementation provides an example of how a modified CFIR framework could be used for IR in humanitarian contexts. This mapping exercise found factors that mapped to all five of the original CFIR domains as well as the sixth domain proposed by Means et al. All factors fit well within the definition for the selected domain, indicating an appropriate degree of fit between these existing domains and the factors identified as impacting SRH interventions in humanitarian settings. On a construct level, however, the findings were more variable, with one-quarter of factors not fully aligning with any construct. Furthermore, over 40% of the CFIR constructs (including the additional constructs from Means et al.) were not found to align with any factors cited by the papers in this review, also demonstrating some disconnect between the parameters posed by the CFIR constructs and the factors cited as relevant in a humanitarian context.

It is worth noting that while the CFIR as proposed in 2009 was used in this assessment, as well as in the included papers which used the CFIR, an update was published in 2022. Following a review of CFIR use since its publication, the authors provide updates to construct names and definitions to “make the framework more applicable across a range of innovations and settings.” New constructs and subconstructs were also added, for a total of 48 constructs and 19 subconstructs across the five domains [ 90 ]. A CFIR Outcomes Addendum was also published in 2022, based on recommendations for the CFIR to add outcomes and intended to be used as a complement to the CFIR determinants framework [ 91 ]. These expansions to the CFIR framework may improve applicability of the CFIR in humanitarian settings. Several constructs added to the Outer Setting domain could be of particular utility – critical incidents, local attitudes, and local conditions, each of which could help account for unique challenges faced in contexts of crisis. Sub-constructs added within the Inner Setting domain that seek to clarify structural characteristics and available resources would also be of high utility based on mapping of the factors identified in this review to the original CFIR constructs. As outcomes were not formally included in the CFIR until the 2022 addendum, a separate assessment of implementation outcomes was not undertaken in this review. However, analysis of the factors cited by papers in this review as affecting implementation was derived from the full text of the papers and thus captures content relevant to implementation determinants that is contained within the outcomes.

Given the demonstrated need for additional flexibility within an IR framework for humanitarian contexts, while not a focus of this review, it is worth considering whether a different framework could provide a better fit than the CFIR. Other frameworks have differing points of emphasis that would create different opportunities for flexibility but that do not seem to resolve the challenges experienced in applying the CFIR to a humanitarian context. As one example, the EPIS (Exploration, Preparation, Implementation, Sustainment) Framework considers the impact of inner and outer context on each of four implementation phases; while the constructs within this framework are broader than the CFIR, an emphasis on the intervention characteristics is missing, a domain where stronger alignment within the CFIR is also needed [ 92 ]. Alternatively, the PRISM (Practical, Robust Implementation and Sustainability Model) framework is a determinant and evaluation framework that adds consideration of context factors to the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) outcomes framework. It has a stronger emphasis on intervention aspects, with sub-domains to account for both organization and patient perspectives within the intervention. While PRISM does include aspects of context, external environment considerations are less robust and intentionally less comprehensive in scope, which would not provide the degree of alignment possible between the Characteristics of Systems and Outer Setting CFIR domains for the considerations unique to humanitarian environments [ 93 ].

Reflecting on their experience with the CFIR, Sarker et al. indicate that it can be a “great asset” in both evaluating current work and developing future interventions. They also encourage future research of humanitarian health interventions to utilize the CFIR [ 51 ]. The other papers that used the CFIR do not specifically reflect on their experience utilizing it, referring more generally to having felt that it was a useful tool [ 65 , 70 ]. On their use of an evaluation framework, Berg et al. reflected that it lent useful structure and helped to identify aspects affecting implementation that otherwise would have gone un-noticed [ 67 ]. The remaining studies that utilized an IR framework did not specifically comment on their experience with its use [ 72 , 83 ]. While a formal IR framework was not engaged by other studies, a number cite a desire for IR to contribute further detail to their findings [ 21 , 37 ].

In their recommendations for strengthening the evidence base for humanitarian health interventions, Ager et al. speak to the need for “methodologic innovation” to develop methodologies with particular applicability in humanitarian settings [ 7 ]. As IR is not yet routinized for SRH interventions, this could be opportune timing for the use of a standardized IR framework to gauge its utility. Using an IR framework to assess factors influencing implementation of the MISP in initial stages of a humanitarian response, and interventions to support more comprehensive SRH service delivery in protracted crises, could lend further rigor and standardization to SRH evaluations, as well as inform strategies to improve MISP implementation over time. Based on categorizing factors identified by these papers as relevant for intervention evaluation, there does seem to be utility to a modified CFIR approach. Given the paucity of formal IR framework use within SRH literature, it would be worth conducting similar scoping exercises to assess for explicit use of IR frameworks within the evidence base for other health service delivery areas in humanitarian settings. In the interim, the recommended approach from this review for future IR on humanitarian health interventions would be a modified CFIR approach with domain-level standardization and flexibility for constructs that may standardize over time with more use. This would enable use of a common analytical framework and vocabulary at the domain level for stakeholders to describe interventions and the factors influencing the effectiveness of implementation, with constructs available to use and customize as most appropriate for specific contexts and interventions.

This review had a number of limitations. As this was a scoping review and a two-part search strategy was used, the papers summarized here may not be comprehensive of those written pertaining to SRH interventions over the past 10 years. Papers from 2013 to 2017 that would have met this scoping review’s inclusion criteria may have been omitted due to being excluded from the systematic reviews. The review was limited to papers available in English. Furthermore, this review did not assess the quality of the papers included or seek to assess the methodology used beyond examination of the use of an IR framework. It does, however, serve as a first step in assessing the extent to which calls for implementation research have been addressed, and identify entry points for strengthening the science and practice of SRH research in humanitarian settings.

With one in 23 people worldwide in need of humanitarian assistance, and financing required for response plans at an all-time high, the need for evidence to guide resource allocation and programming for SRH in humanitarian settings is as important as ever [ 94 ]. Recent research agenda setting initiatives and strategies to advance health in humanitarian settings call for increased investment in implementation research—with priorities ranging from research on effective strategies for expanding access to a full range of contraceptive options to integrating mental health and psychosocial support into SRH programming to capturing accurate and actionable data on maternal and perinatal mortality in a wide range of acute and protracted emergency contexts [ 95 , 96 ]. To truly advance guidance in these areas, implementation research will need to be conducted across diverse humanitarian settings, with clear and consistent documentation of both intervention characteristics and outcomes, as well as contextual and programmatic factors affecting implementation.

Conclusions

Implementation research has potential to increase impact of health interventions particularly in crisis-affected settings where flexibility, adaptability and context-responsive approaches are highlighted as cornerstones of effective programming. There remains significant opportunity for standardization of research in the humanitarian space, with one such opportunity occurring through increased utilization of IR frameworks such as a modified CFIR approach. Investing in more robust sexual and reproductive health research in humanitarian contexts can enrich insights available to guide programming and increase transferability of learning across settings.

Availability of data and materials

The datasets analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Availability, Accessibility, Acceptability and Quality

Centers for Disease Control and Prevention

Consolidated Framework for Implementation Research

Democratic Republic of the Congo

Exploration, Preparation, Implementation, Sustainment

• Implementation research

Low and middle income country

Minimum Initial Service Package

Practical, Robust Implementation and Sustainability Model

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Reach, Effectiveness, Adoption, Implementation, Maintenance

• Sexual and reproductive health

World Health Organization

Peters DH, et al. Implementation research: what it is and how to do it. RESEARCH METHODS. 2013;347:7.

Means AR, et al. Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use in low- and middle-income countries: a systematic review. Implement Sci. 2020;15(1):17.

Article   PubMed   PubMed Central   Google Scholar  

Alonge O, et al. How is implementation research applied to advance health in low-income and middle-income countries? BMJ Glob Health. 2019;4(2):e001257.

Ridde V, Pérez D, Robert E. Using implementation science theories and frameworks in global health. BMJ Glob Health. 2020;5(4):e002269.

Gaffey MF, et al. Delivering health and nutrition interventions for women and children in different conflict contexts: a framework for decision making on what, when, and how. Lancet (London, England). 2021;397(10273):543–54.

Article   PubMed   Google Scholar  

Singh NS, et al. Delivering health interventions to women, children, and adolescents in conflict settings: what have we learned from ten country case studies? The Lancet. 2021;397(10273):533–42.

Article   Google Scholar  

Ager A, et al. Strengthening the evidence base for health programming in humanitarian crises. Science. 2014;345(6202):1290–2.

Article   CAS   PubMed   Google Scholar  

Blanchet K, et al. Evidence on public health interventions in humanitarian crises. The Lancet. 2017;390(10109):2287–96.

Sphere A. The Sphere Handbook | Standards for quality humanitarian response. 2018.

Google Scholar  

Barot S. In a State of Crisis: Meeting the Sexual and Reproductive Health Needs of Women in Humanitarian Situations. Guttmacher Policy Rev. 2017;20:7.

Crisis, I.-A.W.G.f.R.H.i., Minimum Initial Service Package. 2020: https://www.unfpa.org/resources/minimum-initial-service-package-misp-srh-crisis-situations .

Casey SE. Evaluations of reproductive health programs in humanitarian settings: a systematic review. Confl Heal. 2015;9(1):S1.

Singh NS, et al. A long way to go: a systematic review to assess the utilisation of sexual and reproductive health services during humanitarian crises. BMJ Glob Health. 2018;3(2):e000682.

Singh NS, et al. Evaluating the effectiveness of sexual and reproductive health services during humanitarian crises: A systematic review. PLoS ONE. 2018;13(7):e0199300.

Warren E, et al. Systematic review of the evidence on the effectiveness of sexual and reproductive health interventions in humanitarian crises. BMJ Open. 2015;5(12):e008226.

Dadich A, Piper A, Coates D. Implementation science in maternity care: a scoping review. Implement Sci. 2021;16(1):16.

Tricco AC, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73.

Devine A, et al. Strategies for the prevention of perinatal hepatitis B transmission in a marginalized population on the Thailand-Myanmar border: a cost-effectiveness analysis. BMC Infect Dis. 2017;17(1):552.

Jarrett P, et al. Evaluation of a population mobility, mortality, and birth surveillance system in South Kivu. Democratic Republic of the Congo Disasters. 2020;44(2):390–407.

PubMed   Google Scholar  

Logie CH, et al. A Psycho-Educational HIV/STI Prevention Intervention for Internally Displaced Women in Leogane, Haiti: Results from a Non-Randomized Cohort Pilot Study. PLoS ONE. 2014;9(2):e89836.

O’Laughlin KN, et al. A cohort study to assess a communication intervention to improve linkage to HIV care in Nakivale Refugee Settlement. Uganda Glob Public Health. 2021;16(12):1848–55.

Adam I. The influence of maternal health education on the place of delivery in conflict settings of Darfur. Sudan Conflict and Health. 2015;9:31.

Adam IF, et al. Relationship between implementing interpersonal communication and mass education campaigns in emergency settings and use of reproductive healthcare services: evidence from Darfur, Sudan. BMJ Open. 2015;5(9):e008285.

Edmond K, et al. Mobile outreach health services for mothers and children in conflict-affected and remote areas: a population-based study from Afghanistan. Arch Dis Child. 2020;105(1):18–25.

Nasir S, et al. Dissemination and implementation of the e-MCHHandbook, UNRWA’s newly released maternal and child health mobile application: a cross-sectional study. BMJ Open. 2020;10(3):e034885.

O’Laughlin KN, et al. Feasibility and acceptability of home-based HIV testing among refugees: a pilot study in Nakivale refugee settlement in southwestern Uganda. BMC Infect Dis. 2018;18(1):332.

Adam I. Evidence from cluster surveys on the association between home-based counseling and use of family planning in conflict-affected Darfur. Int J Gynecol Obstet. 2016;133(2):221–5.

Casey S, et al. Availability of long-acting and permanent family-planning methods leads to increase in use in conflict-affected northern Uganda: Evidence from cross-sectional baseline and endline cluster surveys. Glob Public Health. 2013;8(3):284–97.

Corna F, et al. Supporting maternal mental health of Rohingya refugee women during the perinatal period to promote child health and wellbeing: a field study in Cox’s Bazar. Intervention, the Journal of Mental Health & Psychosocial Support in Conflict Affected Areas. 2019;17(2):160–8.

Glass N, et al. Effectiveness of the Communities Care programme on change in social norms associated with gender-based violence (GBV) with residents in intervention compared with control districts in Mogadishu, Somalia. BMJ Open. 2019;9(3):e023819.

James LE, et al. Development and Testing of a Community-Based Intervention to Address Intimate Partner Violence among Rohingya and Syrian Refugees: A Social Norms-Based Mental Health-Integrated Approach. Int J Environ Res Public Health. 2021;18(21):11674.

Le Roux E, et al. Engaging with faith groups to prevent VAWG in conflict-affected communities: results from two community surveys in the DRC. BMC Int Health Hum Rights. 2020;20(1):27.

Morris CN, et al. When political solutions for acute conflict in Yemen seem distant, demand for reproductive health services is immediate: a programme model for resilient family planning and post-abortion care services. Sex Reprod Health Matters. 2019;27(2):1610279.

Anibueze AU, et al. Impact of counseling visual multimedia on use of family planning methods among displaced Nigerian families. Health Promot Int. 2022;37(3):daac060.

Doocy S, et al. Cash-based assistance and the nutrition status of pregnant and lactating women in the Somalia food crisis: A comparison of two transfer modalities. PLoS ONE. 2020;15(4):e0230989.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Draiko CV, et al. The effect of umbilical cord cleansing with chlorhexidine gel on neonatal mortality among the community births in South Sudan: a quasi-experimental study. Pan Afr Med J. 2021;38:78.

Edmond KM, et al. Can community health worker home visiting improve care-seeking and maternal and newborn care practices in fragile states such as Afghanistan? A population-based intervention study. BMC Med. 2018;16(1):106.

Edmond KM, et al. Conditional cash transfers to improve use of health facilities by mothers and newborns in conflict affected countries, a prospective population based intervention study from Afghanistan. BMC Pregnancy Childbirth. 2019;19(1):193.

Bakesiima R, et al. Effect of peer counselling on acceptance of modern contraceptives among female refugee adolescents in northern Uganda: A randomised controlled trial. PLoS ONE. 2021;16(9):e0256479.

Greene MC, et al. Evaluation of an integrated intervention to reduce psychological distress and intimate partner violence in refugees: Results from the Nguvu cluster randomized feasibility trial. PLoS ONE. 2021;16(6):e0252982.

Gupta J, et al. Gender norms and economic empowerment intervention to reduce intimate partner violence against women in rural Côte d’Ivoire: a randomized controlled pilot study. BMC Int Health Hum Rights. 2013;13(1):46.

Hossain M, et al. Working with men to prevent intimate partner violence in a conflict-affected setting: a pilot cluster randomized controlled trial in rural Côte d’Ivoire. BMC Public Health. 2014;14(1):339.

Vaillant J, et al. Engaging men to transform inequitable gender attitudes and prevent intimate partner violence: a cluster randomised controlled trial in North and South Kivu, Democratic Republic of Congo. BMJ Glob Health. 2020;5(5):e002223.

Thommesen T, et al. “The midwife helped me … otherwise I could have died”: women’s experience of professional midwifery services in rural Afghanistan - a qualitative study in the provinces Kunar and Laghman. BMC Pregnancy Childbirth. 2020;20(1):140.

Awasom-Fru A, et al. Doctors’ experiences providing sexual and reproductive health care at Catholic Hospitals in the conflict-affected North-West region of Cameroon: a qualitative study. Reprod Health. 2022;19(1):126.

Kabakian-Khasholian T, Makhoul J, Ghusayni A. “A person who does not have money does not enter”: a qualitative study on refugee women’s experiences of respectful maternity care. BMC Pregnancy and Childbirth. 2022;22(1):748.

Lilleston P, et al. Evaluation of a mobile approach to gender-based violence service delivery among Syrian refugees in Lebanon. Health Policy Plan. 2018;33(7):767–76.

Mugo NS, et al. Barriers Faced by the Health Workers to Deliver Maternal Care Services and Their Perceptions of the Factors Preventing Their Clients from Receiving the Services: A Qualitative Study in South Sudan. Matern Child Health J. 2018;22(11):1598–606.

Persson M, et al. A qualitative study on health care providers’ experiences of providing comprehensive abortion care in Cox’s Bazar, Bangladesh. Conflict and Health. 2021;15(1):6.

Phanwichatkul T, et al. The perceptions and practices of Thai health professionals providing maternity care for migrant Burmese women: An ethnographic study. Women Birth. 2022;35(4):e356–68.

Sarker M, et al. Effective maternal, newborn and child health programming among Rohingya refugees in Cox’s Bazar, Bangladesh: Implementation challenges and potential solutions. PLoS ONE. 2020;15(3):e0230732.

Tousaw E, et al. “Without this program, women can lose their lives”: migrant women’s experiences with the Safe Abortion Referral Programme in Chiang Mai. Thailand Reprod Health Matters. 2017;25(51):58–68.

Tousaw E, et al. “It is just like having a period with back pain”: exploring women’s experiences with community-based distribution of misoprostol for early abortion on the Thailand-Burma border. Contraception. 2018;97(2):122–9.

West L, et al. Factors in use of family planning services by Syrian women in a refugee camp in Jordan. Journal of Family Planning and Reproductive Health Care. 2017;43(2):96–102.

O’Connell KA, et al. Meeting the Sexual and Reproductive Health Needs of Internally Displaced Persons in Ethiopia’s Somali Region: A Qualitative Process Evaluation. Glob Health Sci Pract. 2022;10(5):e2100818.

Orya E, et al. Strengthening close to community provision of maternal health services in fragile settings: an exploration of the changing roles of TBAs in Sierra Leone and Somaliland. BMC Health Serv Res. 2017;17(1):460.

Perera SM, et al. Barriers to seeking post-abortion care in Paktika Province, Afghanistan: a qualitative study of clients and community members. BMC Womens Health. 2021;21(1):390.

Tanabe M, et al. Piloting community-based medical care for survivors of sexual assault in conflict-affected Karen State of eastern Burma. Confl Heal. 2013;7(1):12.

Tran NT, et al. Clinical outreach refresher trainings in crisis settings (S-CORT): clinical management of sexual violence survivors and manual vacuum aspiration in Burkina Faso, Nepal, and South Sudan. Reprod Health Matters. 2017;25(51):103–13.

Yankah E, et al. Feasibility and acceptability of mobile phone platforms to deliver interventions to address gender-based violence among Syrian adolescent girls and young women in Izmir. Turkey Vulnerable Children and Youth Studies. 2020;15(2):133–43.

Muuo S, et al. Barriers and facilitators to care-seeking among survivors of gender-based violence in the Dadaab refugee complex. Sex Reprod Health Matters. 2020;28(1):1722404.

Amsalu R, et al. Essential newborn care practice at four primary health facilities in conflict affected areas of Bossaso, Somalia: a cross-sectional study. Conflict and Health. 2019;13(13):27.

Myers A, et al. Facilitators and barriers in implementing the Minimum Initial Services Package (MISP) for reproductive health in Nepal post-earthquake. Conflict and Health. 2018;12:35.

Santo L.C.d, et al. Feasibility and acceptability of a video library tool to support community health worker counseling in rural Afghan districts: a cross-sectional assessment. Conflict and Health. 2020;14:56.

Sami S, et al. Understanding health systems to improve community and facility level newborn care among displaced populations in South Sudan: a mixed methods case study. BMC Pregnancy Childbirth. 2018;18(1):325.

Amsalu R, et al. Effectiveness of clinical training on improving essential newborn care practices in Bossaso, Somalia: a pre and postintervention study. BMC Pediatr. 2020;20(1):215.

Berg M, Mwambali SN, Bogren M. Implementation of a three-pillar training intervention to improve maternal and neonatal healthcare in the Democratic Republic Of Congo: a process evaluation study in an urban health zone. Glob Health Action. 2022;15(1):2019391.

Castillo M, et al. Turning Disaster into an Opportunity for Quality Improvement in Essential Intrapartum and Newborn Care Services in the Philippines: Pre- to Posttraining Assessments. Biomed Res Int. 2016;2016:1–9.

Foster AM, Arnott G, Hobstetter M. Community-based distribution of misoprostol for early abortion: evaluation of a program along the Thailand-Burma border. Contraception. 2017;96(4):242–7.

Guan TH, et al. Implementation of a neonatal hepatitis B immunization program in rural Karenni State, Myanmar: A mixed-methods study. PLoS ONE. 2021;16(12):e0261470.

Logie, C.H., et al., Mixed-methods findings from the Ngutulu Kagwero (agents of change) participatory comic pilot study on post-rape clinical care and sexual violence prevention with refugee youth in a humanitarian setting in Uganda. Global Public Health, 2022((Logie C.H., [email protected]) Factor-Inwentash Faculty of Social Work, University of Toronto, Toronto, Canada(Logie C.H., [email protected]) Women’s College Research Institute, Women’s College Hospital, Toronto, Canada(Logie C.H., carmen.l).

Sami S, et al. “You have to take action”: changing knowledge and attitudes towards newborn care practices during crisis in South Sudan. Reprod Health Matters. 2017;25(51):124–39.

Smith JR, et al. Clinical care for sexual assault survivors multimedia training: a mixed-methods study of effect on healthcare providers’ attitudes, knowledge, confidence, and practice in humanitarian settings. Confl Heal. 2013;7(1):14.

Stevens A, et al. Folate supplementation to prevent birth abnormalities: evaluating a community-based participatory action plan for refugees and migrant workers on the Thailand-Myanmar border. Public Health. 2018;161:83–9.

Nguyen Toan T, et al. Strengthening healthcare providers’ capacity for safe abortion and postabortion care services in humanitarian settings: lessons learned from the clinical outreach refresher training model (S-CORT) in Uganda, Nigeria, and the Democratic Republic of Congo. Conflict and Health. 2021;15(1):20.

Castle S, et al. Successful programmatic approaches to facilitating IUD uptake: CARE’s experience in DRC. BMC Womens Health. 2019;19(1):104.

Deitch J, et al. “They Love Their Patients”: Client Perceptions of Quality of Postabortion Care in North and South Kivu, the Democratic Republic of the Congo. Global health, science and practice. 2019;7(Suppl 2):S285–98.

Ferreyra C, et al. Evaluation of a community-based HIV test and start program in a conflict affected rural area of Yambio County, South Sudan. PLoS ONE. 2021;16(7):e0254331.

Ho LS, Wheeler E. Using Program Data to Improve Access to Family Planning and Enhance the Method Mix in Conflict-Affected Areas of the Democratic Republic of the Congo. Glob Health Sci Pract. 2018;6(1):161–77.

Klabbers RE, et al. Health Worker Perspectives on Barriers and Facilitators of Assisted Partner Notification for HIV for Refugees and Ugandan Nationals: A Mixed Methods Study in West Nile Uganda. AIDS Behav. 2021;25(10):3206–22.

Turner C, et al. Neonatal Intensive Care in a Karen Refugee Camp: A 4 Year Descriptive Study. PLoS ONE. 2013;8(8):e72721.

Vries Id, et al. Key lessons from a mixed-method evaluation of a postnatal home visit programme in the humanitarian setting of Gaza. Eastern Mediterr Health J. 2021;27(6):546–52.

Bolan NE, et al. mLearning in the Democratic Republic of the Congo: A Mixed-Methods Feasibility and Pilot Cluster Randomized Trial Using the Safe Delivery App. Global health, science and practice. 2018;6(4):693–710.

Khan MN, et al. Evaluating feasibility and acceptability of a local psycho-educational intervention for pregnant women with common mental problems affected by armed conflict in Swat, Pakistan: A parallel randomized controlled feasibility trial. Int J Soc Psychiatry. 2017;63(8):724–35.

Hynes M, et al. Using a quality improvement approach to improve maternal and neonatal care in North Kivu, Democratic Republic of Congo. Reprod Health Matters. 2017;25(51):140–50.

Gibbs A, et al. The impacts of combined social and economic empowerment training on intimate partner violence, depression, gender norms and livelihoods among women: an individually randomised controlled trial and qualitative study in Afghanistan. BMJ Glob Health. 2020;5(3):e001946.

Damschroder L, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation science: IS; 2009.

Moore GF, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

Proctor E, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health. 2011;38(2):65–76.

Damschroder LJ, et al. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.

Damschroder LJ, et al. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implement Sci. 2022;17(1):7.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(1):4–23.

Feldstein AC, Glasgow RE. A Practical, Robust Implementation and Sustainability Model (PRISM) for Integrating Research Findings into Practice. The Joint Commission Journal on Quality and Patient Safety. 2008;34(4):228–43.

OCHA. Global Humanitarian Overview 2023. 2022 [cited 2023 8/3/2023]; Available from: https://humanitarianaction.info/node/13073/article/glance-0 . Accessed 8 Mar 2023.

Kobeissi L, et al. Setting research priorities for sexual, reproductive, maternal, newborn, child and adolescent health in humanitarian settings. Confl Heal. 2021;15(1):16.

Save the, C., et al. Roadmap to Accelerate Progress for Every Newborn in Humanitarian Settings 2020 – 2024. 2020. p. 52.

Inter-Agency Working Group on Reproductive Health in, C. Inter-Agency Field Manual on Reproductive Health in Humanitarian Settings. 2018.

Download references

Acknowledgements

Not applicable.

The authors received no funding for this study.

Author information

Authors and affiliations.

Duke University School of Medicine, 40 Duke Medicine Circle, Durham, NC, 27710, USA

Alexandra Norton

Johns Hopkins Bloomberg School of Public Health, 615 N. Wolfe St, Baltimore, MD, 21205, USA

Hannah Tappis

You can also search for this author in PubMed   Google Scholar

Contributions

AN and HT designed the scoping review. AN conducted the literature search. AN and HT screened records for inclusion. AN extracted data from included studies. Both authors contributed to synthesis of results. AN drafted the manuscript and both authors contributed to editorial changes.

Corresponding author

Correspondence to Alexandra Norton .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

. Literature search terms: Exact search terms used in literature search, with additional detail on the methodology to determine search terms and definitions used for each component of the search

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Norton, A., Tappis, H. Sexual and reproductive health implementation research in humanitarian contexts: a scoping review. Reprod Health 21 , 64 (2024). https://doi.org/10.1186/s12978-024-01793-2

Download citation

Received : 06 November 2023

Accepted : 12 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1186/s12978-024-01793-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Humanitarian settings

Reproductive Health

ISSN: 1742-4755

• General enquiries: [email protected]