Author

Sebastian Elbaum

188 papers · 1999–2026

1999peak 15 in 20082026

Frequent co-authors

Papers

showing 1–25 of 188 RSS

Test case prioritization: a family of empirical studies

Sebastian Elbaum, Alexey Malishevsky, Gregg Rothermel

Software · IEEE Transactions on Software Engineering · 2002 · cited 938× · DOI ↗ · more like this ·

To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of such prioritization is to increase a test suite's rate of fault detection. Previous work reported results of studies that showed that prioritization techniques can significantly improve rate of fault detection. Those studies, however, raised several additional questions: 1) Can prioritization techniques be effective when targeted at specific modified versions; 2) what trade-offs exist between fine granularity and coarse granularity prioritization techniques; 3) can the incorporation of measures of fault proneness into prioritization techniques improve their effectiveness? To address these questions, we have performed several new studies in which we empirically compared…

Techniques for improving regression testing in continuous integration development environments

Sebastian Elbaum, Gregg Rothermel, John Penix

Software · 4 Nov 2014 · cited 326× · DOI ↗ · more like this ·

In continuous integration development environments, software engineers frequently integrate new or changed code with the mainline codebase. This can reduce the amount of code rework that is needed as systems evolve and speed up development time. While continuous integration processes traditionally require that extensive testing be performed following the actual submission of code to the codebase, it is also important to ensure that enough testing is performed prior to code submission to avoid breaking builds and delaying the fast feedback that makes continuous integration desirable. In this work, we present algorithms that make continuous integration processes more cost-effective. In an initial pre-submit phase of testing, developers specify modules to be tested, and we use regression test selection techniques to select a subset of the test suites for those modules that render that…

Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization

Sebastian Elbaum, Alexey Malishevsky, Gregg Rothermel

Information Systems · Lincoln (University of Nebraska) · 2001 · cited 322× · OpenAlex ↗ · more like this ·

Test case prioritization techniques schedule test cases for regression testing in an order that increases their ability to meet some performance goal. One performance goal, rate of fault detection, measures how quickly faults are detected within the testing process. In previous work we provided a metric, APFD, for measuring rate of fault detection, and techniques for prioritizing test cases to improve APFD, and reported the results of experiments using those techniques. This metric and these techniques, however, applied only in cases in which test costs and fault severity are uniform. In this paper, we present a new metric for assessing the rate of fault detection of prioritized test cases, that incorporates varying test case and fault costs. We present the results of a case study illustrating the application of the metric. This study raises several practical questions that might arise…

Prioritizing test cases for regression testing

Sebastian Elbaum, Alexey Malishevsky, Gregg Rothermel

Software · Lincoln (University of Nebraska) · August 2000 · cited 322× · DOI ↗ · more like this ·

Test case prioritization techniques schedule test cases in an order that increases their effectiveness in meeting some performance goal. One performance goal, rate of fault detection, is a measure of how quickly faults are detected within the testing process; an improved rate of fault detection can provide faster feedback on the system under test, and let software engineers begin locating and correcting faults earlier than might otherwise be possible. In previous work, we reported the results of studies that showed that prioritization techniques can significantly improve rate of fault detection. Those studies, however, raised several additional questions: (1) can prioritization techniques be effective when aimed at specific modified versions; (2) what tradeoffs exist between fine granularity and coarse granularity prioritization techniques; (3) can the incorporation of measures of fault…

Differential symbolic execution

Suzette Person, Matthew B. Dwyer, Sebastian Elbaum +1 more

Information Systems · Lincoln (University of Nebraska) · 9 Nov 2008 · cited 269× · DOI ↗ · more like this ·

Detecting and characterizing the effects of software changes is a fundamental component of software maintenance. Version differencing information can be used to perform version merging, infer change characteristics, produce program documentation, and guide program re-validation. Existing techniques for characterizing code changes, however, are imprecise leading to unnecessary maintenance efforts.

How developers search for code: a case study

Caitlin Sadowski, Kathryn T. Stolee, Sebastian Elbaum

Information Systems · 26 Aug 2015 · cited 240× · DOI ↗ · more like this ·

With the advent of large code repositories and sophisticated search capabilities, code search is increasingly becoming a key software development activity. In this work we shed some light into how developers search for code through a case study performed at Google, using a combination of survey and log-analysis methodologies. Our study provides insights into what developers are doing and trying to learn when per- forming a search, search scope, query properties, and what a search session under different contexts usually entails. Our results indicate that programmers search for code very frequently, conducting an average of five search sessions with 12 total queries each workday. The search queries are often targeted at a particular code location and programmers are typically looking for code with which they are somewhat familiar. Further, programmers are generally seeking answers to…

Leveraging user-session data to support Web application testing

Sebastian Elbaum, Gregg Rothermel, S. Karre +1 more

Software · IEEE Transactions on Software Engineering · March 2005 · cited 215× · DOI ↗ · more like this ·

Web applications are vital components of the global information infrastructure, and it is important to ensure their dependability. Many techniques and tools for validating Web applications have been created, but few of these have addressed the need to test Web application functionality and none have attempted to leverage data gathered in the operation of Web applications to assist with testing. In this paper, we present several techniques for using user session data gathered as users operate Web applications to help test those applications from a functional standpoint. We report results of an experiment comparing these new techniques to existing white-box techniques for creating test cases for Web applications, assessing both the adequacy of the generated test cases and their ability to detect faults on a point-of-sale Web application. Our results show that user session data can be used…

Improving Web Application Testing with User Session Data

Sebastian Elbaum, Srikanth Karre, Gregg Rothermel

Software · 2003 · cited 180× · OpenAlex ↗ · more like this ·

Web applications have become critical components of the global information infrastructure, and it is important that they be validated to ensure their reliability. Therefore, many techniques and tools for validating web applications have been created. Only a few of these techniques, however, have addressed problems of testing the functionality of web applications, and those that do have not fully considered the unique attributes of web applications. In this paper we explore the notion that user session data gathered as users operate web applications can be successfully employed in the testing of those applications, particularly as those applications evolve and experience different usage profiles. We report results of an experiment comparing new and existing test generation techniques for web applications, assessing both the adequacy of the generated tests and their ability to detect…

Programmers' build errors: a case study (at google)

Hyunmin Seo, Caitlin Sadowski, Sebastian Elbaum +2 more

Information Systems · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · 20 May 2014 · cited 174× · DOI ↗ · more like this ·

Building is an integral part of the software development process. However, little is known about the compiler errors that occur in this process. In this paper, we present an empirical study of 26.6 million builds produced during a period of nine months by thousands of developers. We describe the workflow through which those builds are generated, and we analyze failure frequency, compiler error types, and resolution efforts to fix those compiler errors. The results provide insights on how a large organization build process works, and pinpoints errors for which further developer support would be most effective.

Code churn: a measure for estimating the impact of code change

John C. Munson, Sebastian Elbaum

Information Systems · 27 Nov 2002 · cited 152× · DOI ↗ · more like this ·

This study presents a methodology that will produce a viable fault surrogate. The focus of the effort is on the precise measurement of software development process and product outcomes. Tools and processes for the static measurement of the source code have been installed and made operational in a large embedded software system. Source code measurements have been gathered unobtrusively for each build in the software evolution process. The measurements are synthesized to obtain the fault surrogate. The complexity of sequential builds is compared and a new measure, code churn, is calculated. This paper demonstrates the effectiveness of code complexity churn by validating it against the testing problem reports.

Predicting accurate and actionable static analysis warnings

Joseph R. Ruthruff, John Penix, J. David Morgenthaler +2 more

Information Systems · 10 May 2008 · cited 125× · DOI ↗ · more like this ·

Static analysis tools report software defects that may or may not be detected by other verification methods. Two challenges complicating the adoption of these tools are spurious false positive warnings and legitimate warnings that are not acted on. This paper reports automated support to help address these challenges using logistic regression models that predict the foregoing types of warnings from signals in the warnings and implicated code. Because examining many potential signaling factors in large software development settings can be expensive, we use a screening methodology to quickly discard factors with low predictive power and cost-effectively build predictive models. Our empirical evaluation indicates that these models can achieve high accuracy in predicting accurate and actionable static analysis warnings, and suggests that the models are competitive with alternative models…

Solving the Search for Source Code

Kathryn T. Stolee, Sebastian Elbaum, Daniel Dobos

Information Systems · ACM Transactions on Software Engineering and Methodology · May 2014 · cited 124× · DOI ↗ · more like this ·

Programmers frequently search for source code to reuse using keyword searches. The search effectiveness in facilitating reuse, however, depends on the programmer's ability to specify a query that captures how the desired code may have been implemented. Further, the results often include many irrelevant matches that must be filtered manually. More semantic search approaches could address these limitations, yet existing approaches are either not flexible enough to find approximate matches or require the programmer to define complex specifications as queries. We propose a novel approach to semantic code search that addresses several of these limitations and is designed for queries that can be described using a concrete input/output example. In this approach, programmers write lightweight specifications as inputs and expected output examples. Unlike existing approaches to semantic search…

On test suite composition and cost-effective regression testing

Gregg Rothermel, Sebastian Elbaum, Alexey Malishevsky +2 more

Software · ACM Transactions on Software Engineering and Methodology · July 2004 · cited 114× · DOI ↗ · more like this ·

Regression testing is an expensive testing process used to revalidate software as it evolves. Various methodologies for improving regression testing processes have been explored, but the cost-effectiveness of these methodologies has been shown to vary with characteristics of regression test suites. One such characteristic involves the way in which test inputs are composed into test cases within a test suite. This article reports the results of controlled experiments examining the effects of two factors in test suite composition---test suite granularity and test input grouping---on the costs and benefits of several regression-testing-related methodologies: retest-all, regression test selection, test suite reduction, and test case prioritization. These experiments consider the application of several specific techniques, from each of these methodologies, across ten releases each of two…

Context-Aware Adaptive Applications: Fault Patterns and Their Automated Identification

Michele Sama, Sebastian Elbaum, Franco Raimondi +2 more

Computer Networks and Communications · IEEE Transactions on Software Engineering · 19 Mar 2010 · cited 96× · DOI ↗ · more like this ·

Applications running on mobile devices are intensely context-aware and adaptive. Streams of context values continuously drive these applications, making them very powerful but, at the same time, susceptible to undesired configurations. Such configurations are not easily exposed by existing validation techniques, thereby leading to new analysis and testing challenges. In this paper, we address some of these challenges by defining and applying a new model of adaptive behavior called an Adaptation Finite-State Machine (A-FSM) to enable the detection of faults caused by both erroneous adaptation logic and asynchronous updating of context information, with the latter leading to inconsistencies between the external physical context and its internal representation within an application. We identify a number of adaptation fault patterns, each describing a class of faulty behaviors. Finally, we…

Profiling deployed software: assessing strategies and testing opportunities

Sebastian Elbaum, Madeline Diep

Software · IEEE Transactions on Software Engineering · April 2005 · cited 91× · DOI ↗ · more like this ·

An understanding of how software is employed in the field can yield many opportunities for quality improvements. Profiling released software can provide such an understanding. However, profiling released software is difficult due to the potentially large number of deployed sites that must be profiled, the transparency requirements at a user's site, and the remote data collection and deployment management process. Researchers have recently proposed various approaches to tap into the opportunities offered by profiling deployed systems and overcome those challenges. Initial studies have illustrated the application of these approaches and have shown their feasibility. Still, the proposed approaches, and the tradeoffs between overhead, accuracy, and potential benefits for the testing activity have been barely quantified. This paper aims to overcome those limitations. Our analysis of 1,200…

Carving differential unit test cases from system test cases

Sebastian Elbaum, Hui Nee Chin, Matthew B. Dwyer +1 more

Software · Defense Technical Information Center (DTIC) · 5 Nov 2006 · cited 89× · DOI ↗ · more like this ·

Unit test cases are focused and efficient. System tests are effective at exercising complex usage patterns. Differential unit tests (DUT) are a hybrid of unit and system tests. They are generated by carving the system components, while executing a system test case, that influence the behavior of the target unit, and then re-assembling those components so that the unit can be exercised as it was by the system test. We conjecture that DUTs retain some of the advantages of unit tests, can be automatically and inexpensively generated, and have the potential for revealing faults related to intricate system executions. In this paper we present a framework for automatically carving and replaying DUTs that accounts for a wide-variety of strategies, we implement an instance of the framework with several techniques to mitigate test cost and enhance flexibility, and we empirically assess the…

Parallel Randomized State-Space Search

Matthew B. Dwyer, Sebastian Elbaum, Suzette Person +1 more

Software · Proceedings/Proceedings - International Conference on Software Engineering · May 2007 · cited 87× · DOI ↗ · more like this ·

Model checkers search the space of possible program behaviors to detect errors and to demonstrate their absence. Despite major advances in reduction and optimization techniques, state-space search can still become cost-prohibitive as program size and complexity increase. In this paper, we present a technique for dramatically improving the cost- effectiveness of state-space search techniques for error detection using parallelism. Our approach can be composed with all of the reduction and optimization techniques we are aware of to amplify their benefits. It was developed based on insights gained from performing a large empirical study of the cost-effectiveness of randomization techniques in state-space analysis. We explain those insights and our technique, and then show through a focused empirical study that our technique speeds up analysis by factors ranging from 2 to over 1000 as…

Automated Generation of Context-Aware Tests

Zhimin Wang, Sebastian Elbaum, David S. Rosenblum

Computer Vision and Pattern Recognition · UCL Discovery (University College London) · May 2007 · cited 86× · DOI ↗ · more like this ·

The incorporation of context-awareness capabilities into pervasive applications allows them to leverage contextual information to provide additional services while maintaining an acceptable quality of service. These added capabilities, however, introduce a distinct input space that can affect the behavior of these applications at any point during their execution, making their validation quite challenging. In this paper, we introduce an approach to improve the test suite of a context-aware application by identifying context-aware program points where context changes may affect the application's behavior, and by systematically manipulating the context data fed into the application to increase its exposure to potentially valuable context variations. Preliminary results indicate that the approach is more powerful than existing testing approaches used on this type of application.

Understanding the effects of changes on the cost‐effectiveness of regression testing techniques

Sebastian Elbaum, Praveen Kallakuri, Alexey Malishevsky +2 more

Software · Software Testing Verification and Reliability · April 2003 · cited 86× · DOI ↗ · more like this ·

Abstract Regression testing is an expensive testing process used to validate modified software. Regression test selection and test‐case prioritization can reduce the costs of regression testing by selecting a subset of test cases for execution, or scheduling test cases to meet testing objectives better. The cost‐effectiveness of these techniques can vary widely, however, and one cause of this variance is the type and magnitude of changes made in producing a new software version. Engineers unaware of the causes and effects of this variance can make poor choices in designing change integration processes, selecting inappropriate regression testing techniques, designing excessively expensive regression test suites and making unnecessarily costly changes. Engineers aware of causal factors can perform regression testing more cost‐effectively. This article reports the results of an embedded…

Modeling the cost-benefits tradeoffs for regression testing techniques

Alexey Malishevsky, Gregg Rothermel, Sebastian Elbaum

Software · 26 Jun 2003 · cited 85× · DOI ↗ · more like this ·

Regression testing is an expensive activity that can account for a large proportion of the software maintenance budget. Because engineers add tests into test suites as software evolves, over time, increased test suite size makes revalidation of the software more expensive. Regression test selection, test suite reduction, and test case prioritization techniques can help with this, by reducing the number of regression tests that must be run and by helping testers meet testing objectives more quickly. These techniques, however can be expensive to employ and may not reduce overall regression testing costs. Thus, practitioners and researchers could benefit from cost models that would help them assess the cost-benefits of techniques. Cost models have been proposed for this purpose, but some of these models omit important factors, and others cannot truly evaluate cost-effectiveness. In this…

Automatic generation of load tests

Pingyu Zhang, Sebastian Elbaum, Matthew B. Dwyer

Software · Lincoln (University of Nebraska) · November 2011 · cited 79× · DOI ↗ · more like this ·

Load tests aim to validate whether system performance is acceptable under peak conditions. Existing test generation techniques induce load by increasing the size or rate of the input. Ignoring the particular input values, however, may lead to test suites that grossly mischaracterize a system's performance. To address this limitation we introduce a mixed symbolic execution based approach that is unique in how it 1) favors program paths associated with a performance measure of interest, 2) operates in an iterative-deepening beam-search fashion to discard paths that are unlikely to lead to high-load tests, and 3) generates a test suite of a given size and level of diversity. An assessment of the approach shows it generates test suites that induce program response times and memory consumption several times worse than the compared alternatives, it scales to large and complex inputs, and it…

Exploring the use of crowdsourcing to support empirical studies in software engineering

Kathryn T. Stolee, Sebastian Elbaum

Computer Science Applications · 14 Sep 2010 · cited 76× · DOI ↗ · more like this ·

The power and the generality of the findings obtained through empirical studies are bounded by the number and type of participating subjects. In software engineering, obtaining a large number of adequate subjects to evaluate a technique or tool is often a major challenge. In this work we explore the use of crowdsourcing as a mechanism to address that challenge by assisting in subject recruitment. More specifically, through this work we show how we adapted a study to be performed under an infrastructure that not only makes it possible to reach a large base of users but it also provides capabilities to manage those users as the study is being conducted. We discuss the lessons we learned through this experience, which illustrate the potential and tradeoffs of crowdsourcing software engineering studies.

Bug Hunt: Making Early Software Testing Lessons Engaging and Affordable

Sebastian Elbaum, Suzette Person, Jon Dokulil +1 more

Software · May 2007 · cited 73× · DOI ↗ · more like this ·

Software testing efforts account for a large part of software development costs. However, as educators, we struggle to properly prepare students to perform software testing activities. This struggle is caused by multiple factors: (1) it is challenging to effectively incorporate software testing into an already over-packed curriculum, (2) ad-hoc efforts to teach testing generally happen too late in the students' career, after bad habits have already been developed, and (3) these efforts lack the necessary institutional consistency and support to be effective. To address these challenges we created Bug Hunt, a web-based tutorial to engage students in learning software testing strategies. In this paper we describe the most interesting aspects of the tutorial including the lessons and feedback mechanisms, and the facilities for instructors to configure the tutorial and obtain automatic…

The impact of software evolution on code coverage information

Sebastian Elbaum, D. Gable, Gregg Rothermel

Information Systems · Lincoln (University of Nebraska) · 13 Nov 2002 · cited 72× · DOI ↗ · more like this ·

Many tools and techniques for addressing software maintenance problems rely on code coverage information. Often, this coverage information is gathered for a specific version of a software system, and then used to perform analyses on subsequent versions of that system without being recalculated. As a software system evolves, however, modifications to the software alter the software's behavior on particular inputs, and code coverage information gathered on earlier versions of a program may not accurately reflect the coverage that would be obtained on later versions. This discrepancy may affect the success of analyses dependent on code coverage information. Despite the importance of coverage information in various analyses, in our search of the literature we find no studies specifically examining the impact of software evolution on code coverage information. Therefore, we conducted…

Redefining prioritization

Jingjing Liang, Sebastian Elbaum, Gregg Rothermel

Software · 27 May 2018 · cited 71× · DOI ↗ · more like this ·

Continuous integration (CI) development environments allow software engineers to frequently integrate and test their code. While CI environments provide advantages, they also utilize non-trivial amounts of time and resources. To address this issue, researchers have adapted techniques for test case prioritization (TCP) to CI environments. To date, however, the techniques considered have operated on test suites, and have not achieved substantial improvements. Moreover, they can be inappropriate to apply when system build costs are high. In this work we explore an alternative: prioritization of commits. We use a lightweight approach based on test suite failure and execution history that is highly efficient; our approach "continuously" prioritizes commits that are waiting for execution in response to the arrival of each new commit and the completion of each previously scheduled commit. We…