software engineering research papers pdf

• Search by keyword
• Search by citation

Page 1 of 2

Metric-centered and technology-independent architectural views for software comprehension

The maintenance of applications is a crucial activity in the software industry. The high cost of this process is due to the effort invested on software comprehension since, in most of cases, there is no up-to-...

• View Full Text

Back to the future: origins and directions of the “Agile Manifesto” – views of the originators

In 2001, seventeen professionals set up the manifesto for agile software development. They wanted to define values and basic principles for better software development. On top of being brought into focus, the ...

Investigating the effectiveness of peer code review in distributed software development based on objective and subjective data

Code review is a potential means of improving software quality. To be effective, it depends on different factors, and many have been investigated in the literature to identify the scenarios in which it adds qu...

On the benefits and challenges of using kanban in software engineering: a structured synthesis study

Kanban is increasingly being used in diverse software organizations. There is extensive research regarding its benefits and challenges in Software Engineering, reported in both primary and secondary studies. H...

Challenges on applying genetic improvement in JavaScript using a high-performance computer

Genetic Improvement is an area of Search Based Software Engineering that aims to apply evolutionary computing operators to the software source code to improve it according to one or more quality metrics. This ...

Actor’s social complexity: a proposal for managing the iStar model

Complex systems are inherent to modern society, in which individuals, organizations, and computational elements relate with each other to achieve a predefined purpose, which transcends individual goals. In thi...

Investigating measures for applying statistical process control in software organizations

The growing interest in improving software processes has led organizations to aim for high maturity, where statistical process control (SPC) is required. SPC makes it possible to analyze process behavior, pred...

An approach for applying Test-Driven Development (TDD) in the development of randomized algorithms

TDD is a technique traditionally applied in applications with deterministic algorithms, in which the input and the expected result are known. However, the application of TDD with randomized algorithms have bee...

Supporting governance of mobile application developers from mining and analyzing technical questions in stack overflow

There is a need to improve the direct communication between large organizations that maintain mobile platforms (e.g. Apple, Google, and Microsoft) and third-party developers to solve technical questions that e...

Working software over comprehensive documentation – Rationales of agile teams for artefacts usage

Agile software development (ASD) promotes working software over comprehensive documentation. Still, recent research has shown agile teams to use quite a number of artefacts. Whereas some artefacts may be adopt...

Development as a journey: factors supporting the adoption and use of software frameworks

From the point of view of the software framework owner, attracting new and supporting existing application developers is crucial for the long-term success of the framework. This mixed-methods study explores th...

Applying user-centered techniques to analyze and design a mobile application

Techniques that help in understanding and designing user needs are increasingly being used in Software Engineering to improve the acceptance of applications. Among these techniques we can cite personas, scenar...

A measurement model to analyze the effect of agile enterprise architecture on geographically distributed agile development

Efficient and effective communication (active communication) among stakeholders is thought to be central to agile development. However, in geographically distributed agile development (GDAD) environments, it c...

A survey of search-based refactoring for software maintenance

This survey reviews published materials related to the specific area of Search-Based Software Engineering that concerns software maintenance and, in particular, refactoring. The survey aims to give a comprehen...

Guest editorial foreword for the special issue on automated software testing: trends and evidence

Similarity testing for role-based access control systems.

Access control systems demand rigorous verification and validation approaches, otherwise, they can end up with security breaches. Finite state machines based testing has been successfully applied to RBAC syste...

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

Combinatorial Interaction Testing (CIT) approaches have drawn attention of the software testing community to generate sets of smaller, efficient, and effective test cases where they have been successful in det...

How diverse is your team? Investigating gender and nationality diversity in GitHub teams

Building an effective team of developers is a complex task faced by both software companies and open source communities. The problem of forming a “dream”

Investigating factors that affect the human perception on god class detection: an analysis based on a family of four controlled experiments

Evaluation of design problems in object oriented systems, which we call code smells, is mostly a human-based task. Several studies have investigated the impact of code smells in practice. Studies focusing on h...

On the evaluation of code smells and detection tools

Code smells refer to any symptom in the source code of a program that possibly indicates a deeper problem, hindering software maintenance and evolution. Detection of code smells is challenging for developers a...

On the influence of program constructs on bug localization effectiveness

Software projects often reach hundreds or thousands of files. Therefore, manually searching for code elements that should be changed to fix a failure is a difficult task. Static bug localization techniques pro...

DyeVC: an approach for monitoring and visualizing distributed repositories

Software development using distributed version control systems has become more frequent recently. Such systems bring more flexibility, but also greater complexity to manage and monitor multiple existing reposi...

A genetic algorithm based framework for software effort prediction

Several prediction models have been proposed in the literature using different techniques obtaining different results in different contexts. The need for accurate effort predictions for projects is one of the ...

Elaboration of software requirements documents by means of patterns instantiation

Studies show that problems associated with the requirements specifications are widely recognized for affecting software quality and impacting effectiveness of its development process. The reuse of knowledge ob...

ArchReco: a software tool to assist software design based on context aware recommendations of design patterns

This work describes the design, development and evaluation of a software Prototype, named ArchReco, an educational tool that employs two types of Context-aware Recommendations of Design Patterns, to support us...

On multi-language software development, cross-language links and accompanying tools: a survey of professional software developers

Non-trivial software systems are written using multiple (programming) languages, which are connected by cross-language links. The existence of such links may lead to various problems during software developmen...

SoftCoDeR approach: promoting Software Engineering Academia-Industry partnership using CMD, DSR and ESE

The Academia-Industry partnership has been increasingly encouraged in the software development field. The main focus of the initiatives is driven by the collaborative work where the scientific research work me...

Issues on developing interoperable cloud applications: definitions, concepts, approaches, requirements, characteristics and evaluation models

Among research opportunities in software engineering for cloud computing model, interoperability stands out. We found that the dynamic nature of cloud technologies and the battle for market domination make clo...

Game development software engineering process life cycle: a systematic review

Software game is a kind of application that is used not only for entertainment, but also for serious purposes that can be applicable to different domains such as education, business, and health care. Multidisc...

Correlating automatic static analysis and mutation testing: towards incremental strategies

Traditionally, mutation testing is used as test set generation and/or test evaluation criteria once it is considered a good fault model. This paper uses mutation testing for evaluating an automated static anal...

A multi-objective test data generation approach for mutation testing of feature models

Mutation approaches have been recently applied for feature testing of Software Product Lines (SPLs). The idea is to select products, associated to mutation operators that describe possible faults in the Featur...

An extended global software engineering taxonomy

In Global Software Engineering (GSE), the need for a common terminology and knowledge classification has been identified to facilitate the sharing and combination of knowledge by GSE researchers and practition...

A systematic process for obtaining the behavior of context-sensitive systems

Context-sensitive systems use contextual information in order to adapt to the user’s current needs or requirements failure. Therefore, they need to dynamically adapt their behavior. It is of paramount importan...

Distinguishing extended finite state machine configurations using predicate abstraction

Extended Finite State Machines (EFSMs) provide a powerful model for the derivation of functional tests for software systems and protocols. Many EFSM based testing problems, such as mutation testing, fault diag...

Extending statecharts to model system interactions

Statecharts are diagrams comprised of visual elements that can improve the modeling of reactive system behaviors. They extend conventional state diagrams with the notions of hierarchy, concurrency and communic...

On the relationship of code-anomaly agglomerations and architectural problems

Several projects have been discontinued in the history of the software industry due to the presence of software architecture problems. The identification of such problems in source code is often required in re...

An approach based on feature models and quality criteria for adapting component-based systems

Feature modeling has been widely used in domain engineering for the development and configuration of software product lines. A feature model represents the set of possible products or configurations to apply i...

Patch rejection in Firefox: negative reviews, backouts, and issue reopening

Writing patches to fix bugs or implement new features is an important software development task, as it contributes to raise the quality of a software system. Not all patches are accepted in the first attempt, ...

Investigating probabilistic sampling approaches for large-scale surveys in software engineering

Establishing representative samples for Software Engineering surveys is still considered a challenge. Specialized literature often presents limitations on interpreting surveys’ results, mainly due to the use o...

Characterising the state of the practice in software testing through a TMMi-based process

The software testing phase, despite its importance, is usually compromised by the lack of planning and resources in industry. This can risk the quality of the derived products. The identification of mandatory ...

Self-adaptation by coordination-targeted reconfigurations

A software system is self-adaptive when it is able to dynamically and autonomously respond to changes detected either in its internal components or in its deployment environment. This response is expected to ensu...

Templates for textual use cases of software product lines: results from a systematic mapping study and a controlled experiment

Use case templates can be used to describe functional requirements of a Software Product Line. However, to the best of our knowledge, no efforts have been made to collect and summarize these existing templates...

F3T: a tool to support the F3 approach on the development and reuse of frameworks

Frameworks are used to enhance the quality of applications and the productivity of the development process, since applications may be designed and implemented by reusing framework classes. However, frameworks ...

NextBug: a Bugzilla extension for recommending similar bugs

Due to the characteristics of the maintenance process followed in open source systems, developers are usually overwhelmed with a great amount of bugs. For instance, in 2012, approximately 7,600 bugs/month were...

Assessing the benefits of search-based approaches when designing self-adaptive systems: a controlled experiment

The well-orchestrated use of distilled experience, domain-specific knowledge, and well-informed trade-off decisions is imperative if we are to design effective architectures for complex software-intensive syst...

Revealing influence of model structure and test case profile on the prioritization of test cases in the context of model-based testing

Test case prioritization techniques aim at defining an order of test cases that favor the achievement of a goal during test execution, such as revealing failures as earlier as possible. A number of techniques ...

A metrics suite for JUnit test code: a multiple case study on open source software

The code of JUnit test cases is commonly used to characterize software testing effort. Different metrics have been proposed in literature to measure various perspectives of the size of JUnit test cases. Unfort...

Designing fault-tolerant SOA based on design diversity

Over recent years, software developers have been evaluating the benefits of both Service-Oriented Architecture (SOA) and software fault tolerance techniques based on design diversity. This is achieved by creat...

Method-level code clone detection through LWH (Light Weight Hybrid) approach

Many researchers have investigated different techniques to automatically detect duplicate code in programs exceeding thousand lines of code. These techniques have limitations in finding either the structural o...

The problem of conceptualization in god class detection: agreement, strategies and decision drivers

The concept of code smells is widespread in Software Engineering. Despite the empirical studies addressing the topic, the set of context-dependent issues that impacts the human perception of what is a code sme...

• Editorial Board
• Sign up for article alerts and news from this journal

software engineering Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Identifying Non-Technical Skill Gaps in Software Engineering Education: What Experts Expect But Students Don’t Learn

As the importance of non-technical skills in the software engineering industry increases, the skill sets of graduates match less and less with industry expectations. A growing body of research exists that attempts to identify this skill gap. However, only few so far explicitly compare opinions of the industry with what is currently being taught in academia. By aggregating data from three previous works, we identify the three biggest non-technical skill gaps between industry and academia for the field of software engineering: devoting oneself to continuous learning , being creative by approaching a problem from different angles , and thinking in a solution-oriented way by favoring outcome over ego . Eight follow-up interviews were conducted to further explore how the industry perceives these skill gaps, yielding 26 sub-themes grouped into six bigger themes: stimulating continuous learning , stimulating creativity , creative techniques , addressing the gap in education , skill requirements in industry , and the industry selection process . With this work, we hope to inspire educators to give the necessary attention to the uncovered skills, further mitigating the gap between the industry and the academic world.

Opportunities and Challenges in Code Search Tools

Code search is a core software engineering task. Effective code search tools can help developers substantially improve their software development efficiency and effectiveness. In recent years, many code search studies have leveraged different techniques, such as deep learning and information retrieval approaches, to retrieve expected code from a large-scale codebase. However, there is a lack of a comprehensive comparative summary of existing code search approaches. To understand the research trends in existing code search studies, we systematically reviewed 81 relevant studies. We investigated the publication trends of code search studies, analyzed key components, such as codebase, query, and modeling technique used to build code search tools, and classified existing tools into focusing on supporting seven different search tasks. Based on our findings, we identified a set of outstanding challenges in existing studies and a research roadmap for future code search research.

Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines

A meaningful and deep understanding of the human aspects of software engineering (SE) requires psychological constructs to be considered. Psychology theory can facilitate the systematic and sound development as well as the adoption of instruments (e.g., psychological tests, questionnaires) to assess these constructs. In particular, to ensure high quality, the psychometric properties of instruments need evaluation. In this article, we provide an introduction to psychometric theory for the evaluation of measurement instruments for SE researchers. We present guidelines that enable using existing instruments and developing new ones adequately. We conducted a comprehensive review of the psychology literature framed by the Standards for Educational and Psychological Testing. We detail activities used when operationalizing new psychological constructs, such as item pooling, item review, pilot testing, item analysis, factor analysis, statistical property of items, reliability, validity, and fairness in testing and test bias. We provide an openly available example of a psychometric evaluation based on our guideline. We hope to encourage a culture change in SE research towards the adoption of established methods from psychology. To improve the quality of behavioral research in SE, studies focusing on introducing, validating, and then using psychometric instruments need to be more common.

Towards an Anatomy of Software Craftsmanship

Context: The concept of software craftsmanship has early roots in computing, and in 2009, the Manifesto for Software Craftsmanship was formulated as a reaction to how the Agile methods were practiced and taught. But software craftsmanship has seldom been studied from a software engineering perspective. Objective: The objective of this article is to systematize an anatomy of software craftsmanship through literature studies and a longitudinal case study. Method: We performed a snowballing literature review based on an initial set of nine papers, resulting in 18 papers and 11 books. We also performed a case study following seven years of software development of a product for the financial market, eliciting qualitative, and quantitative results. We used thematic coding to synthesize the results into categories. Results: The resulting anatomy is centered around four themes, containing 17 principles and 47 hierarchical practices connected to the principles. We present the identified practices based on the experiences gathered from the case study, triangulating with the literature results. Conclusion: We provide our systematically derived anatomy of software craftsmanship with the goal of inspiring more research into the principles and practices of software craftsmanship and how these relate to other principles within software engineering in general.

On the Reproducibility and Replicability of Deep Learning in Software Engineering

Context: Deep learning (DL) techniques have gained significant popularity among software engineering (SE) researchers in recent years. This is because they can often solve many SE challenges without enormous manual feature engineering effort and complex domain knowledge. Objective: Although many DL studies have reported substantial advantages over other state-of-the-art models on effectiveness, they often ignore two factors: (1) reproducibility —whether the reported experimental results can be obtained by other researchers using authors’ artifacts (i.e., source code and datasets) with the same experimental setup; and (2) replicability —whether the reported experimental result can be obtained by other researchers using their re-implemented artifacts with a different experimental setup. We observed that DL studies commonly overlook these two factors and declare them as minor threats or leave them for future work. This is mainly due to high model complexity with many manually set parameters and the time-consuming optimization process, unlike classical supervised machine learning (ML) methods (e.g., random forest). This study aims to investigate the urgency and importance of reproducibility and replicability for DL studies on SE tasks. Method: In this study, we conducted a literature review on 147 DL studies recently published in 20 SE venues and 20 AI (Artificial Intelligence) venues to investigate these issues. We also re-ran four representative DL models in SE to investigate important factors that may strongly affect the reproducibility and replicability of a study. Results: Our statistics show the urgency of investigating these two factors in SE, where only 10.2% of the studies investigate any research question to show that their models can address at least one issue of replicability and/or reproducibility. More than 62.6% of the studies do not even share high-quality source code or complete data to support the reproducibility of their complex models. Meanwhile, our experimental results show the importance of reproducibility and replicability, where the reported performance of a DL model could not be reproduced for an unstable optimization process. Replicability could be substantially compromised if the model training is not convergent, or if performance is sensitive to the size of vocabulary and testing data. Conclusion: It is urgent for the SE community to provide a long-lasting link to a high-quality reproduction package, enhance DL-based solution stability and convergence, and avoid performance sensitivity on different sampled data.

Predictive Software Engineering: Transform Custom Software Development into Effective Business Solutions

The paper examines the principles of the Predictive Software Engineering (PSE) framework. The authors examine how PSE enables custom software development companies to offer transparent services and products while staying within the intended budget and a guaranteed budget. The paper will cover all 7 principles of PSE: (1) Meaningful Customer Care, (2) Transparent End-to-End Control, (3) Proven Productivity, (4) Efficient Distributed Teams, (5) Disciplined Agile Delivery Process, (6) Measurable Quality Management and Technical Debt Reduction, and (7) Sound Human Development.

Software—A New Open Access Journal on Software Engineering

Software (ISSN: 2674-113X) [...]

Improving bioinformatics software quality through incorporation of software engineering practices

Background Bioinformatics software is developed for collecting, analyzing, integrating, and interpreting life science datasets that are often enormous. Bioinformatics engineers often lack the software engineering skills necessary for developing robust, maintainable, reusable software. This study presents review and discussion of the findings and efforts made to improve the quality of bioinformatics software. Methodology A systematic review was conducted of related literature that identifies core software engineering concepts for improving bioinformatics software development: requirements gathering, documentation, testing, and integration. The findings are presented with the aim of illuminating trends within the research that could lead to viable solutions to the struggles faced by bioinformatics engineers when developing scientific software. Results The findings suggest that bioinformatics engineers could significantly benefit from the incorporation of software engineering principles into their development efforts. This leads to suggestion of both cultural changes within bioinformatics research communities as well as adoption of software engineering disciplines into the formal education of bioinformatics engineers. Open management of scientific bioinformatics development projects can result in improved software quality through collaboration amongst both bioinformatics engineers and software engineers. Conclusions While strides have been made both in identification and solution of issues of particular import to bioinformatics software development, there is still room for improvement in terms of shifts in both the formal education of bioinformatics engineers as well as the culture and approaches of managing scientific bioinformatics research and development efforts.

Inter-team communication in large-scale co-located software engineering: a case study

AbstractLarge-scale software engineering is a collaborative effort where teams need to communicate to develop software products. Managers face the challenge of how to organise work to facilitate necessary communication between teams and individuals. This includes a range of decisions from distributing work over teams located in multiple buildings and sites, through work processes and tools for coordinating work, to softer issues including ensuring well-functioning teams. In this case study, we focus on inter-team communication by considering geographical, cognitive and psychological distances between teams, and factors and strategies that can affect this communication. Data was collected for ten test teams within a large development organisation, in two main phases: (1) measuring cognitive and psychological distance between teams using interactive posters, and (2) five focus group sessions where the obtained distance measurements were discussed. We present ten factors and five strategies, and how these relate to inter-team communication. We see three types of arenas that facilitate inter-team communication, namely physical, virtual and organisational arenas. Our findings can support managers in assessing and improving communication within large development organisations. In addition, the findings can provide insights into factors that may explain the challenges of scaling development organisations, in particular agile organisations that place a large emphasis on direct communication over written documentation.

Aligning Software Engineering and Artificial Intelligence With Transdisciplinary

Study examined AI and SE transdisciplinarity to find ways of aligning them to enable development of AI-SE transdisciplinary theory. Literature review and analysis method was used. The findings are AI and SE transdisciplinarity is tacit with islands within and between them that can be linked to accelerate their transdisciplinary orientation by codification, internally developing and externally borrowing and adapting transdisciplinary theories. Lack of theory has been identified as the major barrier toward towards maturing the two disciplines as engineering disciplines. Creating AI and SE transdisciplinary theory would contribute to maturing AI and SE engineering disciplines.  Implications of study are transdisciplinary theory can support mode 2 and 3 AI and SE innovations; provide an alternative for maturing two disciplines as engineering disciplines. Study’s originality it’s first in SE, AI or their intersections.

Export Citation Format

Share document.

Help | Advanced Search

Computer Science > Software Engineering

Title: the general index of software engineering papers.

Abstract: We introduce the General Index of Software Engineering Papers, a dataset of fulltext-indexed papers from the most prominent scientific venues in the field of Software Engineering. The dataset includes both complete bibliographic information and indexed ngrams (sequence of contiguous words after removal of stopwords and non-words, for a total of 577 276 382 unique n-grams in this release) with length 1 to 5 for 44 581 papers retrieved from 34 venues over the 1971-2020 period.The dataset serves use cases in the field of meta-research, allowing to introspect the output of software engineering research even when access to papers or scholarly search engines is not possible (e.g., due to contractual reasons). The dataset also contributes to making such analyses reproducible and independently verifiable, as opposed to what happens when they are conducted using 3rd-party and non-open scholarly indexing services.The dataset is available as a portable Postgres database dump and released as open data.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

• Publications
• News and Events
• Education and Outreach

Software Engineering Institute

Technical papers.

The SEI Digital Library houses thousands of technical papers and other documents, ranging from SEI Technical Reports on groundbreaking research to conference proceedings, survey results, and source code.

Explainable Verification: Survey, Situations, and New Ideas

April 16, 2024 • white paper, by bjorn andersson, mark h. klein, dionisio de niz.

This report focuses on potential changes in software development practice and research that would help tools used for formal methods explain their output, making software practitioners more likely to trust …

Zero Trust Industry Days 2024 Scenario: Secluded Semiconductors, Inc.

February 27, 2024 • white paper, by rhonda brown.

This scenario guides discussions of solutions submitted to address the challenges of implementing zero trust.

Considerations for Evaluating Large Language Models for Cybersecurity Tasks

February 20, 2024 • white paper, by jeff gennari, shing-hon lau, samuel j. perl, joel parish (openai), girish sastry (openai).

In this paper, researchers from the SEI and OpenAI explore the opportunities and risks associated with using large language models (LLMs) for cybersecurity tasks.

Navigating Capability-Based Planning: The Benefits, Challenges, and Implementation Essentials

February 7, 2024 • white paper, by anandi hira, william nichols.

Based on industry and government sources, this paper summarizes the benefits and challenges of implementing Capability-Based Planning (CBP).

Encoding Verification Arguments to Analyze High-Level Design Certification Claims: Experiment Zero (E0)

January 18, 2024 • white paper, by bjorn andersson, mark h. klein, dionisio de niz, douglas schmidt (vanderbilt university), ronald koontz (boeing company), john lehoczky (carnegie mellon university), george romanski (federal aviation administration), jonathan preston (lockheed martin corporation), daniel shapiro (institute of defense analysis), floyd fazi (lockheed martin corporation), david tate (institute of defense analysis), gordon putsche (the boeing company), hyoseung kim (university of california, riverside).

This paper discusses whether automation of certification arguments can identify problems that occur in real systems.

The Measurement Challenges in Software Assurance and Supply Chain Risk Management

December 22, 2023 • white paper, by nancy r. mead, carol woody, scott hissam.

This paper recommends an approach for developing and evaluating cybersecurity metrics for open source and other software in the supply chain.

Report to the Congressional Defense Committees on National Defense Authorization Act (NDAA) for Fiscal Year 2022 Section 835 Independent Study on Technical Debt in Software-Intensive Systems

December 7, 2023 • technical report, by ipek ozkaya, brigid o'hearn, julie b. cohen, forrest shull.

This independent study of technical debt in software-intensive systems was sent to Congress in December 2023 to satisfy the requirements of NDAA Section 835.

Assessing Opportunities for LLMs in Software Engineering and Acquisition

November 1, 2023 • white paper, by julie b. cohen, james ivers, ipek ozkaya, stephany bellomo, shen zhang.

This white paper examines how decision makers, such as technical leads and program managers, can assess the fitness of large language models (LLMs) to address software engineering and acquisition needs.

Acquisition Security Framework (ASF): Managing Systems Cybersecurity Risk (Expanded Set of Practices)

October 2, 2023 • technical note, by michael s. bandor, charles m. wallen, carol woody, christopher j. alberts.

This framework of practices helps programs coordinate their management of engineering and supply chain risks across the systems lifecycle.

Simulating Realistic Human Activity Using Large Language Model Directives

October 2, 2023 • technical report, by sean huff, thomas g. podnar, dustin d. updyke.

The authors explore how activities generated from the GHOSTS Framework’s NPC client compare to activities produced by GHOSTS’ default behavior and LLMs.

Why Your Software Cost Estimates Change Over Time and How DevSecOps Data Can Help Reduce Cost Risk

September 29, 2023 • white paper, by julie b. cohen.

Early software cost estimates are often off by over 40%; this paper discusses how programs must continually update estimates as more information becomes available.

A Retrospective in Engineering Large Language Models for National Security

By andrew o. mellinger, tyler brooks, shannon gallagher, bryan brown, eric heim, hollen barmer, william nichols, nick winski, nathan m. vanhoudnos, jasmine ratchford, angelique mcdowell, swati rallapalli.

This document discusses the findings, recommendations, and lessons learned from engineering a large language model for national security use cases.

U.S. Leadership in Software Engineering and AI Engineering

August 25, 2023 • white paper, by ipek ozkaya, douglas schmidt (vanderbilt university), forrest shull, john e. robert, erin harper, anita carleton.

A joint SEI/NITRD workshop will advance U.S. national interests through software and AI engineering and accelerate progress across virtually all scientific domains.

A Holistic View of Architecture Definition, Evolution, and Analysis

August 24, 2023 • technical report, by james ivers, sebastián echeverría, rick kazman.

This report focuses on performing architectural decisions and architectural analysis, spanning multiple quality attributes, in a sustainable and ongoing way.

Emerging Technologies: Seven Themes Changing the Future of Software in the DoD

August 24, 2023 • white paper, by scott hissam, shen zhang, michael abad-santos.

This report summarizes the SEI's Emerging Technologies Study (ETS) and identifies seven emerging technologies to watch in software engineering practices and technology.

Demonstrating the Practical Utility and Limitations of ChatGPT Through Case Studies

August 23, 2023 • white paper, by clarence worrell, matthew walsh, alejandro gomez, dominic a. ross.

In this study, SEI researchers conducted four case studies using GPT-3.5 to assess the practical utility of large language models such as ChatGPT.

Software Excellence Through the Agile High Velocity Development℠ Process

July 17, 2023 • technical report, by barti k. perini (ishpi information technologies, inc.), stephen shook (ishpi information technologies, inc.).

The High Velocity Development℠ process earned Ishpi Information Technologies, Inc. the 2023 Watts Humphrey Software Quality Award.

Coding the Future: Recommendations for Defense Software R&D

July 13, 2023 • white paper, by software engineering institute.

This report outlines the key recommendations from the November 2022 workshop "Software as a Modernization Priority."

Engineering of Edge Software Systems: A Report from the November 2022 SEI Workshop on Software Systems at the Edge

June 30, 2023 • white paper, by ipek ozkaya, grace lewis, kevin a. pitstick.

Based on a workshop with thought leaders in the field, this report identifies recommended areas of focus for engineering software systems at the edge.

Software Bill of Materials Framework: Leveraging SBOMs for Risk Reduction

June 14, 2023 • white paper, by carol woody, christopher j. alberts, michael s. bandor, charles m. wallen.

This paper is a Software Bill of Materials (SBOM) Framework that is a starting point for expanding the use of SBOMs for managing software and systems risk.

Generative AI: Key Opportunities and Research Challenges

June 9, 2023 • white paper.

This 2023 workshop report identifies DoD use cases for generative AI and discusses meeting challenges and needs such as investing in guardrails and responsible AI amid a race to capability.

Securing UEFI: An Underpinning Technology for Computing

May 30, 2023 • white paper, by vijay s. sarvepalli.

This paper highlights the technical efforts to secure the UEFI-based firmware that serves as a foundational piece of modern computing environments.

Using Model-Based Systems Engineering (MBSE) to Assure a DevSecOps Pipeline is Sufficiently Secure

May 23, 2023 • technical report, by timothy a. chick, nataliya shevchenko, scott pavetti.

This report describes how analysts can use a model-based systems engineering (MBSE) approach to detect and mitigate cybersecurity risks to a DevSecOps pipeline.

A Strategy for Component Product Lines: Report 2: Specification Modeling for Components in a Component Product Line

May 17, 2023 • special report, by john mcgregor, john j. hudak, sholom g. cohen.

This report introduces the “model chain” concept for specifying a component product line and realizing architecture requirements through the creation–evolution process.

A Strategy for Component Product Lines: Report 3: Component Product Line Governance

May 4, 2023 • special report, by sholom g. cohen, alfred schenker.

This report provides guidance for the community involved with developing and sustaining product lines of components used by the U.S. government.

Program Managers—The DevSecOps Pipeline Can Provide Actionable Data

April 24, 2023 • white paper, by julie b. cohen, bill nichols.

This paper describes the Automated Continuous Estimation for a Pipeline of Pipelines research project, which automates data collection to track program progress.

Zero Trust Industry Day 2022: Areas of Future Research

January 25, 2023 • white paper, by timothy morrow, trista polaski, matthew nicolai.

This paper describes the future research discussed at the 2022 Zero Trust Industry Day event.

Industry Best Practices for Zero Trust Architecture

December 13, 2022 • white paper, by timothy morrow, nathaniel richmond, matthew nicolai.

This paper describes best practices identified during the SEI’s Zero Trust Industry Day 2022, and provides ways to help organizations shift to zero trust.

A Strategy for Component Product Lines: Report 1: Scoping, Objectives, and Rationale

December 8, 2022 • special report, by gabriel moreno, john j. hudak, sholom g. cohen, alfred schenker, john mcgregor.

This report establishes a Component Product Line Strategy to address problems in systematically reusing and integrating components built to conform to component specification models.

Acquisition Security Framework (ASF): Managing Systems Cybersecurity Risk

November 11, 2022 • technical note.

This report provides an overview of the Acquisition Security Framework (ASF), a description of the practices developed thus far, and a plan for completing the ASF body of work.

Zero Trust Industry Day Experience Paper

October 31, 2022 • white paper, by rhonda brown, mary popeck, timothy morrow.

This paper describes the results of the 2022 Zero Trust Industry Day event.

Challenge Development Guidelines for Cybersecurity Competitions

October 27, 2022 • technical report, by dennis m. allen, leena arora, joseph vessella, josh hammerstein, matt kaar, jarrett booz.

This paper draws on the SEI’s experience to provide general-purpose guidelines and best practices for developing effective cybersecurity challenges.

Acquisition Security Framework (ASF): An Acquisition and Supplier Perspective on Managing Software-Intensive Systems’ Cybersecurity Risk

October 4, 2022 • white paper, by carol woody, christopher j. alberts, charles m. wallen, michael s. bandor.

The Acquisition Security Framework (ASF) contains practices that support programs acquiring/building a secure, resilient software-reliant system to manage risks.

Designing Vultron: A Protocol for Multi-Party Coordinated Vulnerability Disclosure (MPCVD)

September 15, 2022 • special report, by allen d. householder.

This report proposes a formal protocol specification for MPCVD to improve the interoperability of both CVD and MPCVD processes.

Common Sense Guide to Mitigating Insider Threats, Seventh Edition

September 7, 2022 • technical report.

The guide describes 22 best practices for mitigating insider threat based on the CERT Division's continued research and analysis of more than 3,000 insider threat cases.

Coordinated Vulnerability Disclosure User Stories

August 25, 2022 • white paper, by art manion, timur d. snoke, vijay s. sarvepalli, jonathan spring, allen d. householder, laurie tyzenhaus, brad runyon, eric hatleback, charles g. yarbrough.

This paper provides user stories to guide the development of a technical protocol and application programming interface for Coordinated Vulnerability Disclosure.

LLVM Intermediate Representation for Code Weakness Identification

July 8, 2022 • white paper, by shannon gallagher, william klieber, david svoboda.

This paper examines whether intermediate representation used in Large Language Models can be useful to indicate the presence of software vulnerabilities.

Digital Engineering Effectiveness

May 19, 2022 • white paper, by alfred schenker, bill nichols, tyler smith (adventium labs, inc.).

This paper explores the reluctance of developers of cyber-physical systems to embrace digital engineering (DE), how DE methods should be tailored to achieve their stakeholders' goals, and how to measure …

A Brief Introduction to the Evaluation of Learned Models for Aerial Object Detection

May 2, 2022 • white paper, by eric heim.

The SEI AI Division assembled guidance on the design, production, and evaluation of machine-learning models for aerial object detection.

Guidance for Tailoring DoD Request for Proposals (RFPs) to Include Modeling

April 27, 2022 • special report, by tom merendino, robert wojcik, julie b. cohen.

This report provides guidance for government program offices that are including digital engineering/modeling requirements into a request for proposal.

Modeling to Support DoD Acquisition Lifecycle Events (Version 1.4)

April 26, 2022 • white paper, by tom merendino, julie b. cohen, robert wojcik.

This document provides suggestions for producing requirement, system, and software models that will be used to support various DoD system acquisition lifecycle events.

Experiences with Deploying Mothra in Amazon Web Services (AWS)

April 26, 2022 • technical report, by daniel ruef, john stogoski, brad powell.

The authors describe development of an at-scale prototype of an on-premises system to test the performance of Mothra in the cloud and provide recommendations for similar deployments.

Extensibility

April 6, 2022 • technical report.

This report summarizes how to systematically analyze a software architecture with respect to a quality attribute requirement for extensibility.

TwinOps: Digital Twins Meets DevOps

March 24, 2022 • technical report, by joe yankel, jerome hugues, anton hristozov, john j. hudak.

This report describes ModDevOps, an approach that bridges model-based engineering and software engineering using DevOps concepts and code generation from models, and TwinOps, a specific ModDevOps pipeline.

March 16, 2022 • Technical Report

By philip bianco, james ivers, sebastián echeverría, rick kazman.

This report summarizes how to systematically analyze a software architecture with respect to a quality attribute requirement for robustness.

An Analysis of How Many Undiscovered Vulnerabilities Remain in Information Systems

March 9, 2022 • white paper, by jonathan spring.

This paper examines the paradigm that the number of undiscovered vulnerabilities is manageably small through the lens of mathematical concepts from the theory of computing.

Using XML to Exchange Floating Point Data

February 10, 2022 • white paper, by john klein.

This paper explains issues of using XML to exchange floating point values, how to address them, and the limits of technology to enforce a correct implementation.

Using Machine Learning to Increase NPC Fidelity

December 1, 2021 • technical report, by dustin d. updyke, thomas g. podnar, geoffrey b. dobson, john yarger.

The authors describe how they used machine learning (ML) modeling to create decision-making preferences for non-player characters (NPCs).

A Prototype Set of Cloud Adoption Risk Factors

October 27, 2021 • white paper, by christopher j. alberts.

Alberts discusses the results of a study to identify a prototype set of risk factors for adopting cloud technologies.

Cloud Security Best Practices Derived from Mission Thread Analysis

September 2, 2021 • technical report, by timothy morrow, donald faatz, nathaniel richmond, angel luis hueca, vincent lapiana.

This report presents practices for secure, effective use of cloud computing and risk reduction in transitioning applications and data to the cloud, and considers the needs of limited-resource businesses.

Accenture: An Automation Maturity Journey

July 29, 2021 • technical report, by rajendra t. prasad (accenture).

This paper describes work in the area of automation that netted Accenture the 2020 Watts Humphrey Software Process Achievement Award.

Planning and Design Considerations for Data Centers

July 19, 2021 • technical note, by lyndsi a. hughes, david sweeney, mark kasunic.

This report shares important lessons learned from establishing small- to mid-size data centers.

Integrating Zero Trust and DevSecOps

July 5, 2021 • white paper, by timothy morrow, geoff sanders, nathaniel richmond, carol woody.

This paper discusses the interdependent strategies of zero trust and DevSecOps in the context of application development.

A State-Based Model for Multi-Party Coordinated Vulnerability Disclosure (MPCVD)

July 1, 2021 • special report, by allen d. householder, jonathan spring.

This report discusses performance indicators that stakeholders in Coordinated Vulnerability Disclosure (CVD) can use to measure its effectiveness.

Human-Centered AI

June 25, 2021 • white paper, by jay palat, matt gaston, frank redner, carol j. smith, tanisha smith, hollen barmer, rachel dzombak.

This white paper discusses Human-Centered AI: systems that are designed to work with, and for, people.

Robust and Secure AI

By rachel dzombak, hollen barmer, eric heim, nathan m. vanhoudnos, tanisha smith, frank redner, matt gaston, jay palat.

This white paper discusses Robust and Secure AI systems: AI systems that reliably operate at expected levels of performance, even when faced with uncertainty and in the presence of danger …

Scalable AI

By jay palat, matt gaston, frank redner, tanisha smith, hollen barmer, rachel dzombak, john wohlbier.

This white paper discusses Scalable AI: the ability of AI algorithms, data, models, and infrastructure to operate at the size, speed, and complexity required for the mission.

The Sector CSIRT Framework: Developing Sector-Based Incident Response Capabilities

June 8, 2021 • technical report, by tracy bills, sharon mudd, justin novak, brittany manley, angel luis hueca, david mcintire.

This framework guides the development and implementation of a sector CSIRT.

Foundation of Cyber Ranges

May 19, 2021 • technical report, by bill reed, dustin d. updyke, geoffrey b. dobson, thomas g. podnar.

This report details the design considerations and execution plan for building high-fidelity, realistic virtual cyber ranges that deliver maximum training and exercise value for cyberwarfare participants.

Software Assurance Guidance and Evaluation (SAGE) Tool

May 3, 2021 • white paper, by robert schiela, ebonie mcneil, luiz antunes, hasan yasar.

The Software Assurance Guidance and Evaluation (SAGE) tool helps an organization assess the security of its systems development and operations practices.

Prioritizing Vulnerability Response: A Stakeholder-Specific Vulnerability Categorization (Version 2.0)

April 30, 2021 • white paper, by jonathan spring, allen d. householder, art manion, vijay s. sarvepalli, eric hatleback, laurie tyzenhaus, madison oliver, charles g. yarbrough.

This paper presents version 2.0 of a testable Stakeholder-Specific Vulnerability Categorization (SSVC) that takes the form of decision trees and that avoids some problems with the Common Vulnerability Scoring System …

Modeling and Validating Security and Confidentiality in System Architectures

March 19, 2021 • technical report, by aaron greenhouse, lutz wrage, jörgen hansson (university of skovde).

This report presents an approach for modeling and validating confidentiality using the Bell–LaPadula security model and the Architecture Analysis & Design Language.

Overview of Practices and Processes of the CMMC 1.0 Assessment Guides (CMMC 1.0)

March 3, 2021 • white paper, by douglas gardner.

This document is intended to help anyone unfamiliar with cybersecurity standards get started with the Department of Defense (DoD) Cybersecurity Maturity Model Certification (CMMC).

Zero Trust: Risks and Research Opportunities

March 1, 2021 • white paper, by geoff sanders, timothy morrow.

This paper describes a zero trust vignette and three mission threads that highlight risks and research areas to consider for zero trust environments.

Artificial Intelligence (AI) and Machine Learning (ML) Acquisition and Policy Implications

February 26, 2021 • white paper, by william e. novak.

This paper reports on a high-level survey of a set of both actual and potential acquisition and policy implications of the use of Artificial Intelligence (AI) and Machine Learning (ML) …

Security Engineering Risk Analysis (SERA) Threat Archetypes

December 16, 2020 • white paper, by carol woody, christopher j. alberts.

This report examines the concept of threat archetypes and how analysts can use them during scenario development.

Loss Magnitude Estimation in Support of Business Impact Analysis

December 15, 2020 • technical report, by brett tucker, daniel j. kambic, david tobar, andrew p. moore.

The authors describe a project to develop an estimation method that yields greater confidence in and improved ranges for estimates of potential cyber loss magnitude.

Emerging Technologies 2020: Six Areas of Opportunity

December 14, 2020 • white paper.

This study seeks to understand what the software engineering community perceives to be key emerging technologies. The six technologies described hold great promise and, in some cases, have already attracted …

Maintainability

December 1, 2020 • technical report, by rick kazman, john klein, james ivers, philip bianco.

This report summarizes how to systematically analyze a software architecture with respect to a quality attribute requirement for maintainability.

Advancing Risk Management Capability Using the OCTAVE FORTE Process

November 17, 2020 • technical note, by brett tucker.

OCTAVE FORTE is a process model that helps organizations evaluate their security risks and use ERM principles to bridge the gap between executives and practitioners.

Analytic Capabilities for Improved Software Program Management

November 2, 2020 • white paper, by christopher miller, david zubrow.

This white paper describes an update to the SEI Quantifying Uncertainty in Early Lifecycle Cost Estimation approach.

AI Engineering for Defense and National Security: A Report from the October 2019 Community of Interest Workshop

October 29, 2020 • special report.

Based on a workshop with thought leaders in the field, this report identifies recommended areas of focus for AI Engineering for Defense and National Security.

NICE Framework Cybersecurity Evaluator

August 20, 2020 • white paper, by christopher herr.

This cybersecurity evaluator is designed to assess members of the cyber workforce within the scope of the NICE Cybersecurity Workforce Framework.

Current Ransomware Threats

August 19, 2020 • white paper, by marisa midler, kyle o'meara.

This report by Marisa Midler, Kyle O'Meara, and Alexandra Parisi discusses ransomware, including an explanation of its design, distribution, execution, and business model.

An Updated Framework of Defenses Against Ransomware

August 18, 2020 • white paper, by timur d. snoke, timothy j. shimeall.

This report, loosely structured around the NIST Cybersecurity Framework, seeks to frame an approach for defending against Ransomware-as-a-Service (RaaS) as well as direct ransomware attacks.

Historical Analysis of Exploit Availability Timelines

August 13, 2020 • white paper, by david warren, jeff chrabaszcz (govini), trent novelly, allen d. householder, jonathan spring.

This paper analyzes when and how known exploits become associated with the vulnerabilities that made them possible.

Architecture Evaluation for Universal Command and Control

August 3, 2020 • white paper, by john klein, harry l. levinson, reed little, jason popowski, philip bianco, patrick donohoe.

The SEI developed an analysis method to assess function allocations in existing C2 systems and reason about design choices and tradeoffs during the design of new C2 systems.

A Risk Management Perspective for AI Engineering

June 10, 2020 • white paper.

This paper describes several steps of OCTAVE FORTE in the context of adopting AI technology.

Attack Surface Analysis - Reduce System and Organizational Risk

June 8, 2020 • white paper, by robert j. ellison, carol woody.

This paper offers system defenders an overview of how threat modeling can provide a systematic way to identify potential threats and prioritize mitigations.

Guide to Implementing DevSecOps for a System of Systems in Highly Regulated Environments

April 8, 2020 • technical report, by jose a. morales, peter capell, david james shepard, richard turner, patrick r. place, suzanne miller.

This Technical Report provides guidance to projects interested in implementing DevSecOps (DSO) in defense or other highly regulated environments, including those involving systems of systems.

Integrability

February 7, 2020 • technical report, by rick kazman, john klein, philip bianco, james ivers.

This report summarizes how to systematically analyze a software architecture with respect to a quality attribute requirement for integrability.

Comments on NISTIR 8269 (A Taxonomy and Terminology of Adversarial Machine Learning)

February 4, 2020 • white paper, by jonathan spring, april galyardt, nathan m. vanhoudnos.

Feedback to the U.S. National Institute of Standards and Technology (NIST) about NIST IR 8269, a draft report detailing the proposed taxonomy and terminology of Adversarial Machine Learning (AML).

Penetration Tests Are The Check Engine Light On Your Security Operations

January 7, 2020 • white paper, by dan j. klinedinst, allen d. householder.

A penetration test serves as a lagging indicator of a network security operations problem. Organizations should implement and document several security controls before a penetration test can be useful.

Prioritizing Vulnerability Response: A Stakeholder-Specific Vulnerability Categorization

December 4, 2019 • white paper, by allen d. householder, jonathan spring, art manion, deana shick, eric hatleback.

This paper presents a testable Stakeholder-Specific Vulnerability Categorization (SSVC) that takes the form of decision trees and that avoids some problems with the Common Vulnerability Scoring System (CVSS).

AI Engineering: 11 Foundational Practices

September 12, 2019 • white paper.

This initial set of recommendations can help organizations that are beginning to build, acquire, and integrate artificial intelligence capabilities into business and mission systems.

Machine Learning in Cybersecurity: A Guide

September 5, 2019 • technical report, by ed stoner, joshua fallon, april galyardt, jonathan spring, leigh b. metcalf, angela horneman.

This report suggests seven key questions that managers and decision makers should ask about machine learning tools to effectively use those tools to solve cybersecurity problems.

Operational Test & Evaluation (OT&E) Roadmap for Cloud-Based Systems

September 2, 2019 • white paper, by john klein, christopher j. alberts, carol woody, charles m. wallen.

This paper provides an overview of the preparation and work that the AEC needs to perform to successfully transition the Army to cloud computing.

IEEE Computer Society/Software Engineering Institute Watts S. Humphrey Software Process Achievement Award 2018: U.S. Army Combat Capabilities Development Command Armaments Center, Fire Control Systems and Technology Directorate

August 1, 2019 • technical report, by victor a. elias (u.s. army ccdc armaments center, fire control systems and technology directorate).

This report presents a systemic approach to software development process improvement and its impact for the U.S. Army Combat Capabilities Development Command Armaments Center, Fire Control Systems and Technology Directorate …

Overview of Risks, Threats, and Vulnerabilities Faced in Moving to the Cloud

July 11, 2019 • technical report, by kelwyn pender, carrie lee (u.s. department of veteran affairs), donald faatz, timothy morrow.

This report, updated in October 2020, examines the changes to risks, threats, and vulnerabilities when applications are deployed to cloud services.

Automatically Detecting Technical Debt Discussions

June 24, 2019 • white paper, by robert nord, ipek ozkaya, zachary kurtz, raghvinder sangwan.

This study introduces (1) a dataset of expert labels of technical debt in developer comments and (2) a classifier trained on those labels.

Multi-Method Modeling and Analysis of the Cybersecurity Vulnerability Management Ecosystem

By allen d. householder, andrew p. moore.

This paper presents modeling and analysis of two critical foundational processes of the cybersecurity vulnerability management ecosystem using a combination of system dynamics and agent-based modeling techniques.

SCAIFE API Definition Beta Version 0.0.2 for Developers

June 14, 2019 • white paper, by ebonie mcneil, lori flynn.

This paper provides the SCAIFE API definition for beta version 0.0.2. SCAIFE is an architecture that supports static analysis alert classification and prioritization.

Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery

May 21, 2019 • white paper.

We present a preliminary report for xBD, a new large-scale dataset for the advancement of change detection and building damage assessment for humanitarian assistance and disaster recovery research.

Integration of Automated Static Analysis Alert Classification and Prioritization with Auditing Tools: Special Focus on SCALe

May 13, 2019 • technical report, by lori flynn, david svoboda, ebonie mcneil, zachary kurtz, derek leung, jiyeon lee (carnegie mellon university).

This report summarizes progress and plans for developing a system to perform automated classification and advanced prioritization of static analysis alerts.

Cybersecurity Career Paths and Progression

May 7, 2019 • white paper, by nicholas giruzzi, marie baker, dennis m. allen, melissa burns.

This paper explores the current state of cybersecurity careers, from the importance of early exposure, to methods of entry into the field, to career progression.

Cybersecurity Talent Identification and Assessment

By dennis m. allen, marie baker, christopher herr.

To help fill cybersecurity roles, this paper explores how organizations identify talent, discusses assessment capabilities, and provides recommendations on recruitment and talent evaluations.

Cybersecurity Careers of the Future

By dennis m. allen.

Using workforce data analysis, this paper identifies key cybersecurity skills the workforce needs to close the cybersecurity workforce gap.

A Targeted Improvement Plan for Service Continuity

April 8, 2019 • technical note, by philip a. scolieri, jeffrey pinckard, robert a. vrtis, andrew f. hoover, gavin jurecko.

Describes how an organization can leverage the results of a Cyber Resilience Review to create a Targeted Improvement Plan for its service continuity management.

Exploring the Use of Metrics for Software Assurance

March 7, 2019 • technical note, by carol woody, robert j. ellison, charlie ryan.

This report proposes measurements for each Software Assurance Framework (SAF) practice that a program can select to monitor and manage the progress it's making toward software assurance.

Common Sense Guide to Mitigating Insider Threats, Sixth Edition

February 27, 2019 • technical report, by sarah miller, tracy cassidy, michael c. theis, daniel l. costa, william r. claycomb, andrew p. moore, randall f. trzeciak.

The guide presents recommendations for mitigating insider threat based on the CERT Division's continued research and analysis of more than 1,500 insider threat cases.

An Approach for Integrating the Security Engineering Risk Analysis (SERA) Method with Threat Modeling

February 6, 2019 • white paper.

This report examines how cybersecurity data generated by a threat modeling method can be integrated into a mission assurance context using the SERA Method.

Infrastructure as Code: Final Report

January 28, 2019 • white paper, by doug reynolds, john klein.

This project explored the feasibility of infrastructure as code, developed prototype tools, populated a model of the deployment architecture, and automatically generated IaC scripts from the model.

Incident Management Capability Assessment

December 19, 2018 • technical report, by samuel j. perl, mark zajicek, robin ruefle, christopher j. alberts, pennie walters, carly l. huth, audrey j. dorofee, david mcintire.

The capabilities presented in this report provide a benchmark of incident management practices.

Program Manager's Guidebook for Software Assurance

December 14, 2018 • special report, by carol woody, timothy a. chick, kenneth nidiffer.

This guidebook helps program managers address the software assurance responsibilities critical in defending software-intensive systems, including mission threads and cybersecurity.

DoD Developer’s Guidebook for Software Assurance

By bill nichols, tom scanlon.

This guidebook helps software developers for DoD programs understand expectations for software assurance and standards and requirements that affect assurance.

Towards Improving CVSS

December 4, 2018 • white paper, by allen d. householder, jonathan spring, deana shick, art manion, eric hatleback.

This paper outlines challenges with the Common Vulnerability Scoring System (CVSS).

GHOSTS in the Machine: A Framework for Cyber-Warfare Exercise NPC Simulation

December 3, 2018 • technical report, by adam d. cerini, benjamin l. earl, thomas g. podnar, geoffrey b. dobson, luke j. osterritter, dustin d. updyke.

This report outlines how the GHOSTS (General HOSTS) framework helps create realism in cyber-warfare simulations and discusses how it was used in a case study.

Composing Effective Software Security Assurance Workflows

October 18, 2018 • technical report, by bill nichols, jim mchale, aaron volkmann, david sweeney, william snavely.

In an effort to determine how to make secure software development more cost effective, the SEI conducted a research study to empirically measure the effects that security tools—primarily automated static …

FedCLASS: A Case Study of Agile and Lean Practices in the Federal Government

October 5, 2018 • special report, by jeff davenport, tamara marshall-keim, linda parker gates, nanette brown.

This study reports the successes and challenges of using Agile and Lean methods and cloud-based technologies in a government software development environment.

Threat Modeling for Cyber-Physical System-of-Systems: Methods Evaluation

September 25, 2018 • white paper, by nataliya shevchenko, carol woody, brent frye.

This paper compares threat modeling methods for cyber-physical systems and recommends which methods (and combinations of methods) to use.

Software Architecture Publications

September 17, 2018 • white paper.

The SEI compiled this bibliography of publications about software architecture as a resource for information about system architecture throughout its lifecycle.

Practical Precise Taint-flow Static Analysis for Android App Sets

August 27, 2018 • white paper, by william klieber, lori flynn, william snavely, michael zheng.

This paper describes how to detect taint flow in Android app sets with a static analysis method that is fast and uses little disk and memory space.

Threat Modeling: A Summary of Available Methods

August 9, 2018 • white paper, by carol woody, nataliya shevchenko, tom scanlon, timothy a. chick, paige o'riordan.

This paper discusses twelve threat modeling methods from a variety of sources that target different parts of the development process.

Navigating the Insider Threat Tool Landscape: Low-Cost Technical Solutions to Jump-Start an Insider Threat Program

July 3, 2018 • white paper, by michael j. albrethsen, derrick spooner, daniel l. costa, george silowash.

This paper explores low cost technical solutions that can help organizations prevent, detect, and respond to insider incidents.

Blacklist Ecosystem Analysis: July - December 2017

Writing good software engineering research papers

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Understanding peer review of software engineering papers

• Published: 17 July 2021
• Volume 26 , article number  103 , ( 2021 )

Cite this article

• Neil A. Ernst   ORCID: orcid.org/0000-0001-5992-2366 1 ,
• Jeffrey C. Carver   ORCID: orcid.org/0000-0002-7824-9151 2 ,
• Daniel Mendez   ORCID: orcid.org/0000-0003-0619-6027 3 , 4 &
• Marco Torchiano   ORCID: orcid.org/0000-0001-5328-368X 5  

692 Accesses

7 Citations

2 Altmetric

Explore all metrics

Peer review is a key activity intended to preserve the quality and integrity of scientific publications. However, in practice it is far from perfect.

We aim at understanding how reviewers, including those who have won awards for reviewing, perform their reviews of software engineering papers to identify both what makes a good reviewing approach and what makes a good paper.

We first conducted a series of interviews with recognised reviewers in the software engineering field. Then, we used the results of those interviews to develop a questionnaire used in an online survey and sent out to reviewers from well-respected venues covering a number of software engineering disciplines, some of whom had won awards for their reviewing efforts.

We analyzed the responses from the interviews and from 175 reviewers who completed the online survey (including both reviewers who had won awards and those who had not). We report on several descriptive results, including: Nearly half of award-winners (45%) are reviewing 20+ conference papers a year, while 28% of non-award winners conduct that many. The majority of reviewers (88%) are taking more than two hours on journal reviews. We also report on qualitative results. Our findings suggest that the most important criteria of a good review is that it should be factual and helpful, which ranked above others such as being detailed or kind. The most important features of papers that result in positive reviews are a clear and supported validation, an interesting problem, and novelty. Conversely, negative reviews tend to result from papers that have a mismatch between the method and the claims and from papers with overly grandiose claims. Further insights include, if not limited to, that reviewers view data availability and its consistency as being important or that authors need to make their contribution of the work very clear in their paper.

Conclusions

Based on the insights we gained through our study, we conclude our work by compiling a proto-guideline for reviewing. One hope we associate with our work is to contribute to the ongoing debate and contemporary effort to further improve our peer review models in the future.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Developers perception of peer code review in research software development

Rapid Reviews in Software Engineering

An Empirical Analysis of Newcomers’ Contributions to Software-Engineering Conferences

http://tiny.cc/rosefest

A good tutorial on data disclosure when using a double-blind review process is provided by Daniel Graziotin: https://tinyurl.com/DBDisclose .

See, e.g., artifact evaluation track of ICSE 2021 https://doi.org/10.6084/m9.gshare.14123639 or the open science initiative of the EMSE journal https://github.com/emsejournal/openscience

https://github.com/acmsigsoft/EmpiricalStandards

https://github.com/researchart/patterns/blob/master/standards/artifact.md

http://www.icse-conferences.org/reports.html

https://neuripsconf.medium.com/what-we-learned-from-neurips-2020-reviewing-process-e24549eea38f

https://cs.gmu.edu/~offutt/stvr/17-3-sept2007.html

https://peerj.com/articles/cs-111/reviews/

https://openreview.net/forum?id=rklXaoAcFX&noteId=HyeF-4_9hm

https://doi.org/10.6084/m9.gshare.5086357.v1

https://www.slideshare.net/aserebrenik/peer-reviews-119010210

https://reviewqualitycollector.org

http://www.inf.fu-berlin.de/w/SE/ReviewQualityCollectorHome

http://cscw.acm.org/2019/CSCW-2020-changes.html

https://conf.researchr.org/track/icse-2021/icse-2021-papers#Call-for-Papers

Drubin DG (2011) Any jackass can trash a manuscript, but it takes good scholarship to create one (how MBoC promotes civil and constructive peer review). Mol Biol Cell 22(5):525–527. https://doi.org/10.1091/mbc.e11-01-0002

Article   Google Scholar  

Fernandes JM (2014) Authorship trends in software engineering. Scientometrics 101(1):257–271. https://doi.org/10.1007/s11192-014-1331-6

Horbach SPJM, Halffman W (2018) The ability of different peer review procedures to ag problematic publications. Scientometrics 118(1):339–373. https://doi.org/10.1007/s11192-018-2969-2

Kitchenham B, Peeger SL (2002) Principles of survey research: part 1-6. ACM SIGSOFT Softw Eng Notes 27(5):17–20

MacAuley D (2012) The role of the manuscript assessor. In: How to write a paper, chap. 16. https://doi.org/10.1002/9781118488713.ch16 . Wiley, pp 102–114

Mathew G, Agrawal A, Menzies T (2019) Finding trends in software research. IEEE Trans Softw Eng :1–1. https://doi.org/10.1109/tse.2018.2870388

Mendez D, Graziotin D, Wagner S, Seibold H (2020) Open science in software engineering. In: Felderer M, Travassos G-H (eds) Contemporary empirical methods in software engineering. arXiv: 1908.05899 . Springer

Nierstrasz O (1998) Identify the champion. Pattern Languages of Programming (PLoP). WUCS-98-25

Ozkaya I (2021) Protecting the health and longevity of the peer-review process in the software engineering community. IEEE Softw 38(1):3–6. https://doi.org/10.1109/ms.2020.3028681

Peterson DAM (2020) Dear reviewer 2: Go F’ Yourself. Soc Sci Q. https://doi.org/10.1111/ssqu.12824

Petre M, et al. (2020) Reviewing computing education papers. In: Proceedings of the 2020 ACM conference on innovation and technology in computer science education. https://doi.org/10.1145/3341525.3394994 . ACM

Prechelt L, Graziotin D, Mendez D (2018) A community’s perspective on the status and future of peer review in software engineering. Inf Softw Technol 95:75–85. https://doi.org/10.1016/j.infsof.2017.10.019

Price E (2014) The NIPS Experiment. http://blog.mrtz.org/2014/12/15/thenipsexperiment.html . Online; accessed August 2020

Ragone A, Mirylenka K, Casati F, Marchese M (2013) On peer review in computer science: analysis of its effectiveness and suggestions for improvement. Scientometrics 97(2):317– 356. https://doi.org/10.1007/s11192-013-1002-z

Ralph P, Robbes R (2020) The ACM SIGSOFT paper and peer review quality initiative. ACM SIGSOFT Softw Eng Notes 45(2):17–18. https://doi.org/10.1145/3385678.3385681

Schimel J (2011) Writing science: how to write papers that get cited and proposals that get funded. Oxford University Press, Oxford

Google Scholar  

Seeber M, Bacchelli A (2017) Does single blind peer review hinder newcomers?. Scientometrics 113(1):567–585. https://doi.org/10.1007/s11192-017-2264-7

Shaw M (2003) Writing good software engineering research papers: minitutorial. In: Proceedings of the ACM/IEEE international conference on software engineering, pp 726–736

Shepperd M, Ajienka N, Counsell S (2018) The role and value of replication in empirical software engineering results. Inf Softw Technol 99:120–132. https://doi.org/10.1016/j.infsof.2018.01.006

Smith AJ (1990) The task of the referee. Computer 23(4):65– 71

Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: CHASE workshop at ICSE

Soldani J, Kuhrmann M, Pfahl D (2020) Pains and gains of peer-reviewing in software engineering. ACM SIGSOFT Softw Eng Notes 45(1):12–13. https://doi.org/10.1145/3375572.3375575

Spier R (2002) The history of the peer-review process. Trends Biotechnol 20 (8):357–358. https://doi.org/10.1016/s0167-7799(02)01985-6

Squazzoni F, Brezis E, Marušić A (2017) Scientometrics of peer review. Scientometrics 113(1):501–502. https://doi.org/10.1007/s11192-017-2518-4.

Storey M-A, Ernst NA, Williams C, Kalliamvakou E (2019) The who, what, how of software engineering research: a socio-technical framework. arXiv: 1905.1284

Tennant JP, et al. (2017) A multi-disciplinary perspective on emergent and future innovations in peer review. In: F1000Research. https://doi.org/10.12688/f1000research.12037.3 , vol 6, p 1151

Terrell J, Kofink A, Middleton J, Rainear C, Murphy-Hill E, Parnin C, Stallings J (2017) Gender differences and bias in open source: pull request acceptance of women versus men. Peer J Comput Sci 3:e111. https://doi.org/10.7717/peerj-cs.111

Theisen C, Dunaiski M, Williams L, Visser W (2017) Writing good software engineering research papers: revisited. In: Proceedings of the ACM/IEEE international conference on software engineering. https://doi.org/10.1109/icse-c.2017.51

Tomkins A, Zhang M, Heavlin WD (2017) Reviewer bias in single- versus double-blind peer review. Proc Natl Acad Sci 114(48):12708–12713. https://doi.org/10.1073/pnas.1707323114

Tung AKH (2006) Impact of double blind reviewing on SIGMOD publication. ACM SIGMOD Record 35 (3):6–7. https://doi.org/10.1145/1168092.1168093

Winslett M, Braganholo V (2016) H V Jagadish speaks out on PVLDB, CoRR and data-driven research. In: SIGMOD Record 42.2

Wolfram D, Wang P, Hembree A, Park H (2020) Open peer review: promoting transparency in open science. Scientometrics 125(2):1033–1051. https://doi.org/10.1007/s11192-020-03488-4

Zerbe WJ, Paulhus DL (1987) Socially desirable responding in organizational behavior: a reconception. Acad Manag Rev 12(2):250. https://doi.org/10.2307/258533

Zong Q, Xie Y, Liang J (2020) Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ. Scientometrics 125 (1):607–623. https://doi.org/10.1007/s11192-020-03545-y.

Download references

Author information

Authors and affiliations.

University of Victoria, Victoria, Canada

Neil A. Ernst

University of Alabama, Tuscaloosa, AL, USA

Jeffrey C. Carver

Blekinge Institute of Technology, Karlskrona, Sweden

Daniel Mendez

fortiss GmbH, Munich, Germany

Politecnico di Torino, Turin, Italy

Marco Torchiano

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Neil A. Ernst .

Additional information

Communicated by: Romain Robbes

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Ernst, N.A., Carver, J.C., Mendez, D. et al. Understanding peer review of software engineering papers. Empir Software Eng 26 , 103 (2021). https://doi.org/10.1007/s10664-021-10005-5

Download citation

Accepted : 16 June 2021

Published : 17 July 2021

DOI : https://doi.org/10.1007/s10664-021-10005-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Peer review
• Find a journal
• Publish with us
• Track your research

Software Engineering

At Google, we pride ourselves on our ability to develop and launch new products and features at a very fast pace. This is made possible in part by our world-class engineers, but our approach to software development enables us to balance speed and quality, and is integral to our success. Our obsession for speed and scale is evident in our developer infrastructure and tools. Developers across the world continually write, build, test and release code in multiple programming languages like C++, Java, Python, Javascript and others, and the Engineering Tools team, for example, is challenged to keep this development ecosystem running smoothly. Our engineers leverage these tools and infrastructure to produce clean code and keep software development running at an ever-increasing scale. In our publications, we share associated technical challenges and lessons learned along the way.

Recent Publications

Some of our teams.

Climate and sustainability

Software engineering and programming languages

We're always looking for more talented, passionate people.

IT Professional Internship

Information for employers, procedure .

• Every year invitations to join the scheme are sent to companies around November.
• The internship period is 9 months and it starts in summer, earliest in July and latest in September.
• Students work in internship 4 days/week, with Fridays off to attend classes at CityU.
• Employers will interview students prior to confirmation of internship, and the company will assign a working supervisor for the students afterwards.
• The performance of the students will be jointly assessed by both academic and working supervisors.

For NEW Internship Company: 

For existing internship company:, roles of internship company.

• Assign work supervisor;
• Initial meeting with academic supervisor;
• Conduct regular student performance assessment (in December & April);
• Market rate trainee/intern salary;
• Welcome to keep close contact with the academic supervisor.

Roles of Our Department

• Assign academic supervisor;
• Initial meeting with work supervisor;
• Support internship company in student coaching & counseling.
• Blank Student Performance Review (Interim) Form  (PDF)
• Blank Student Performance Review (Final) Form  (PDF)

Contact Internship Coordinator

• Email: [email protected]
• Online inquiry form