arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Robert Merkel

36 papers · 2003–2022

2003: 12004: 32005: 22006: 22007: 22008: 12009: 32010: 12011: 42012: 32013: 22014: 42015: 12016: 42017: 02018: 12019: 02020: 12021: 02022: 1
2003peak 4 in 20112022

Frequent co-authors

Papers

showing 1–25 of 36 RSS

Mirror adaptive random testing

Tsong Yueh Chen, Fei‐Ching Kuo, Robert Merkel +1 more

Software · Swinburne Research Bank (Swinburne University of Technology) · · cited 108× · DOI ↗ · more like this ·

Adaptive random testing (ART) has recently been introduced to improve the fault-detection effectiveness of random testing (RT) for certain types of failure-causing patterns. However, ART requires extra computations to ensure an even spread of test cases, which may render ART to be less cost-effective than RT. In this paper, we introduce an innovative approach, namely Mirror Adaptive Random Testing (MART), to reduce these computations. Our simulation results clearly show that MART does improve the cost-effectiveness of ART.

An upper bound on software testing effectiveness

Tsong Yueh Chen, Robert Merkel

Software · ACM Transactions on Software Engineering and Methodology · · cited 99× · DOI ↗ · more like this ·

Failure patterns describe typical ways in which inputs revealing program failure are distributed across the input domain—in many cases, clustered together in contiguous regions. Based on these observations several debug testing methods have been developed. We examine the upper bound of debug testing effectiveness improvements possible through making assumptions about the shape, size and orientation of failure patterns. We consider the bounds for testing strategies with respect to minimizing the F-measure, maximizing the P-measure, and maximizing the E-measure. Surprisingly, we find that the empirically measured effectiveness of some existing methods that are not based on these assumptions is close to the theoretical upper bound of these strategies. The assumptions made to obtain the upper bound, and its further implications, are also examined.

Adaptive random testing through dynamic partitioning

Tsong Yueh Chen, Robert Merkel, Peter Kok-Yiu Wong +1 more

Software · International Conference on Quality Software · · cited 95× · DOI ↗ · more like this ·

Adaptive random testing (ART) describes a family of algorithms for generating random test cases that have been experimentally demonstrated to have greater fault-detection capacity than simple random testing. We outline and demonstrate two new ART algorithms, and demonstrate experimentally that they offer similar performance advantages, with considerably lower overhead than other ART algorithms.

Quasi-Random Testing

Tsong Yueh Chen, Robert Merkel

Hardware and Architecture · IEEE Transactions on Reliability · · cited 59× · DOI ↗ · more like this ·

Our paper proposes an implementable procedure for using the method of quasi-random sequences in software debug testing. In random testing, the sequence of tests (if considered as points in an -dimensional unit hypercube) will give rise to regions where there are clusters of points, as well as underpopulated regions. Quasi-random sequences, also known as low-discrepancy or low-dispersion sequences, are sequences of points in such a hypercube that are spread more evenly throughout. Based on the observation that program faults tend to lead to contiguous failure regions within a program's input domain, and that an even spread of random tests enhances the failure detection effectiveness for certain failure patterns, we examine the use of quasi-random sequences as a replacement for random sequences in automated testing. Because there are only a small number of quasi-random sequence generation…

On the statistical properties of the F-measure

Tsong Yueh Chen, Fei‐Ching Kuo, Robert Merkel

Software · Swinburne Research Bank (Swinburne University of Technology) · · cited 54× · DOI ↗ · more like this ·

The F-measure - the number of distinct test cases to detect the first program failure - is an effectiveness measure for debug testing strategies. We show that for random testing with replacement, the F-measure will be distributed according to the geometric distribution. A simulation study examines the distribution of two adaptive random testing methods, to study how closely their sampling distributions approximate the geometric distribution, revealing that in the worst case scenario, the sampling distribution for adaptive random testing is very similar to random testing. Our results have provided an answer to a conjecture that adaptive random testing is always a more effective alternative to random testing, with reference to the F-measure. We consider the implications of our findings for previous studies conducted in the area, and make recommendations to future studies.

A Cost-Effective Random Testing Method for Programs with Non-Numeric Inputs

Arlinta Barus, Tsong Yueh Chen, Fei‐Ching Kuo +3 more

Software · IEEE Transactions on Computers · · cited 42× · DOI ↗ · more like this ·

Random testing (RT) has been widely used in the testing of various software and hardware systems. Adaptive random testing (ART) is a family of random testing techniques that aim to enhance the failure-detection effectiveness of RT by spreading random test cases evenly throughout the input domain. ART has been empirically shown to be effective on software with numeric inputs. However, there are two aspects of ART that need to be addressed to render its adoption more widespread-applicability to programs with nonnumeric inputs, and the high computation overhead of many ART algorithms. We present a linear-order ART algorithm for software with non-numeric inputs. The key requirement for using ART with non-numeric inputs is an appropriate “distance” measure. We use the concepts of categories and choices from category-partition testing to formulate such a measure. We investigate the…

An empirical study of the effects of personality on software testing

Tanjila Kanij, Robert Merkel, John Grundy

Information Systems · Swinburne Research Bank (Swinburne University of Technology) · · cited 32× · DOI ↗ · more like this ·

The effectiveness of testing is a major determinant of software quality. It is believed that individual testers vary in their effectiveness, but so far the factors contributing to this variation have not been well studied. In this study, we examined whether personality traits, as described by the five-factor model, affect performance on a software testing task. ICT students were given a small software testing task at which their effectiveness was assessed using several different criteria, including bug location rate, weighted fault density, and bug report quality. Their personality was assessed using the NEO PI-3 personality questionnaire. We then compared testing performance according to individual and aggregate measures against different five-factor personality traits. Several weak correlations between two of these personality traits, extraversion and conscientiousness, and testing…

Analysis and enhancements of adaptive random testing

Robert Merkel

Software · Swinburne Research Bank (Swinburne University of Technology) · · cited 31× · DOI ↗ · more like this ·

Random testing is a standard software testing method. It is a popular method for reli-ability assessment, but its use for debug testing has been opposed by some authorities. Random testing does not use any information to guide test case selection, and so, it is argued, testing is less likely to be effective than other methods. Based on the observation that failures often cluster in contiguous regions, Adaptive Random Testing (ART) is a more effective random testing method. While retaining random selection of test cases, selection is guided by the idea that tests should be widely spread throughout the input domain. A simple way to implement this concept, FSCS-ART, involves randomly generating a number of candidates, and choosing the candidate most widely spread from any already-executed test. This method has already shown to be up to 50% more effective than random testing. This thesis…

An empirical investigation of personality traits of software testers

Tanjila Kanij, Robert Merkel, John Grundy

Information Systems · Deakin Research Online (Deakin University) · · cited 24× · DOI ↗ · more like this ·

Abstract—Software testing is the process of an execution-based investigation of some aspects of the software’s quality. The efficiency of the process depends on the methods and technologies used, but crucially also on the human testers. Software testers typically attempt to anticipate and expose ways software may be defective, a fundamentally different task set to those of other software development practitioners. This raises the question of whether the personality of software testers may be different to other people involved in software development. To test this hypothesis, we collected personality profiles using the big five factor model of around 200 software development practitioners. Analysis of this data indicates that software testers are significantly higher on the conscientiousness factor than other software development practitioners, while other factors remain broadly…

A Preliminary Survey of Factors Affecting Software Testers

Tanjila Kanij, Robert Merkel, John Grundy

Software · Deakin Research Online (Deakin University) · · cited 24× · DOI ↗ · more like this ·

Most software testing research has focused on the development of systematic, standardised, and automated testing methodologies and tools. The abilities and expertise needed to apply such techniques and tools - such as personality traits, education, and experience - have attracted a comparatively small amount of research attention. However, the limited research in the area to date provides some indication that the human traits of software testers are important for effective testing. This paper presents the opinions of software testers themselves, collected through an online survey, on the importance of a variety of factors that influence effective testing, including testing-specific training, experience, skills, and human qualities like dedication and general intelligence. The survey responses strongly suggest that while testing tools and training are important, human factors were…

Tag-Based Techniques for Black-Box Test Case Prioritization for Service Testing

Lijun Mei, W. K. Chan, T.H. Tse +1 more

Software · The HKU Scholars Hub (University of Hong Kong) · · cited 23× · DOI ↗ · more like this ·

A web service may evolve autonomously, making peer web services in the same service composition uncertain as to whether the evolved behaviors may still be compatible to its originally collaborative agreement. Although peer services may wish to conduct regression testing to verify the original collaboration, the source code of the former service can be inaccessible to them. Traditional code-based regression testing strategies are inapplicable. The rich interface specifications of a web service, however, provide peer services with a means to formulate black-box testing strategies. In this paper, we formulate new test case prioritization strategies using tags embedded in XML messages to reorder regression test cases, and reveal how the test cases use the interface specifications of services. We evaluate experimentally their effectiveness on revealing regression faults in modified WS-BPEL…

Efficient and effective random testing using the Voronoi diagram

Tsong Yueh Chen, Robert Merkel

Software · Figshare · · cited 22× · DOI ↗ · more like this ·

Adaptive random testing (ART) is a method for improving the fault-finding effectiveness of random testing. Fixed-size candidate set ART is the most studied variant of this approach. However, existing implementations of FSCS-ART have had substantial selection overhead, with n test cases requiring O(n/sup 2/) time to generate. We describe the use of a geometric data structure known as the Voronoi diagram to reduce this overhead to no worse than O(n/spl radic/n) and, with further optimization, O(nlogn). We demonstrate experimentally that practical improvements in selection overhead can be gained using this improved implementation.

Enhanced lattice-based adaptive random testing

Tsong Yueh Chen, De Huang, Fei‐Ching Kuo +2 more

Software · Swinburne Research Bank (Swinburne University of Technology) · · cited 19× · DOI ↗ · more like this ·

Adaptive Random Testing (ART) has been proposed to improve the fault-detection capability of Random Testing (RT). Lattice-based ART (L-ART) is a distinctive ART method which generates test cases by systematically placing and then randomly shifting lattice nodes in the input domain. Previous studies showed that L-ART has a better fault-detection capability than RT, at the same generation cost. Test cases of L-ART however may be highly concentrated on certain parts of the input domain - a "skewed distribution of test cases". Because of this skewed distribution, when failure regions coincidentally reside in the area where L-ART selects a high density of test cases, L-ART can have a better fault-detection capability than when failure regions are in the low density area. Since failure regions can be in any part of the input domain, this dependency of fault-detection capability on the failure…

Technology-Based Responses to Technology-Facilitated Domestic and Family Violence: An Overview of the Limits and Possibilities of Tech-Based “Solutions”

Diarmaid Harkin, Robert Merkel

Sociology and Political Science · Violence Against Women · · cited 11× · DOI ↗ · more like this ·

It is increasingly common for domestic and family violence to have an element of technology-facilitated abuse (TFA). As a result, technology-based responses have emerged to address TFA. Using observations from several empirical research projects into TFA, it will be shown that technology-based responses are necessary without being sufficient, and that they have persistent limitations that need to be recognized. Relatedly, it will be argued that there should be an ongoing emphasis on the development of human resources as a support for those experiencing TFA, particularly the use of professional DV support workers.

Performance assessment metrics for software testers

Tanjila Kanij, Robert Merkel, John Grundy

Software · Deakin Research Online (Deakin University) · · cited 11× · DOI ↗ · more like this ·

Abstract—The reliability of delivered software depends on the performance of the individuals responsible for testing it. However, there are currently no standard methods for assessing the performance of software testers, nor even agreement on im-portant assessment criteria. A literature review reveals several different human-centric factors that have been suggested as important. However, the relative importance of these factors is unknown. This paper reports the results of a survey of professional testers indicating their views on the importance of a number of proposed factors. Their views, we believe, will be important in designing a more formal and widely acceptable performance assessment method for software testers. Keywords-performance, testing, survey, bug count I.

Software Reliability Growth Models Predict Autonomous Vehicle Disengagement Events

Robert Merkel

cs.SE · · cited 9× · arXiv ↗ · more like this ·

The acceptance of autonomous vehicles is dependent on the rigorous assessment of their safety. Furthermore, the commercial viability of AV programs depends on the ability to estimate the time and resources required to achieve desired safety levels. Naive approaches to estimating the reliability and safety levels of autonomous vehicles under development are will require infeasible amounts of testing of a static vehicle configuration. To permit both the estimation of current safety, and make predictions about the reliability of future systems, I propose the use of a standard tool for modelling the reliability of evolving software systems, software reliability growth models (SRGMs). Publicly available data from Californian public-road testing of two autonomous vehicle systems is modelled using two of the best-known SRGMs. The ability of the models to accurately estimate current…

Testing an Open Source Suite for Open Queuing Network Modelling Using Metamorphic Testing Technique

Tsong Yueh Chen, Fei‐Ching Kuo, Robert Merkel +1 more

Software · Swinburne Research Bank (Swinburne University of Technology) · · cited 9× · DOI ↗ · more like this ·

Queuing network modelling is a modelling technique for capacity planning studies of computer and communication systems. Due to complexity of the technique, it is very difficult to know from the computed outputs whether the computation of the modelling software is correct. It is necessary to have an effective testing technique to address this problem. Recently, it has been noticed that metamorphic testing is an effective technique for this kind of problem. In this paper, we study the technique of metamorphic testing in testing queuing network modelling through a set of testing experiments in the Java Modelling Tool – a popular open source queuing network modelling suite.

Dynamic Test Reconfigurationfor Composite Web Services

Mark B. Cooray, James Hamlyn-Harris, Robert Merkel

Software · IEEE Transactions on Services Computing · · cited 7× · DOI ↗ · more like this ·

This paper introduces a testing strategy that is suitable for testing service-based applications. We describe an architecture that responds to changes of service operation, operation arguments and service composition changes. Our proof-of-concept test system performs runtime testing on our model atomic and composite web services using a random testing technique. A novel change identification method was developed to capture changes at the service interface. The test system is able to identify changes that occur in service operations and operational arguments in a service description of a test candidate. Our approach uses a new method to detect changes in a service inventory. Automated reconfiguration is used to support the continuous operation of the testing systems during a test candidate change.

Using the Information: Incorporating Positive Feedback Information into the Testing Process

Kwok Ping Chan, Dave Towey, Tsong Yueh Chen +2 more

Software · Figshare · · cited 7× · DOI ↗ · more like this ·

Software testing is recognized as an essential part of the software development process. Random testing (RT), the selection of input test cases at random from the input domain, is a simple and efficient method of software testing. RT does not however make use of previously executed test case information; in particular, information about nonfailure-causing test cases is ignored. Intuitively, use of this positive feedback information can improve the failure-finding efficiency of a testing method. Adaptive random testing (ART) makes use of knowledge of general failure pattern types, and information of previously executed test cases, in the selection of new test cases. A failure pattern in a program's input domain is composed of the regions of failure-causing inputs. Previous research has categorized failure patterns broadly into three types: point; strip; and block, and has identified…

On the improvement of a fault classification scheme with implications for white-box testing

Shimul Kumar Nath, Robert Merkel, Man Fai Lau

Information Systems · Swinburne Research Bank (Swinburne University of Technology) · · cited 6× · DOI ↗ · more like this ·

Different testing techniques can be more or less effective on different fault types; therefore, testing methods that are most likely to detect the most common fault types should be preferred. However, to enable such a selection of testing methods, a suitable and effective fault classification scheme is essential. Much software testing research proposes techniques to generate test cases and evaluates these techniques based on some hypothesized fault classification scheme. However, there is a lack in the justification of whether such 'hypothesized faults' are realistic and how often these faults occur. Recently, Pan et al. analyzed the syntactic changes in the source code, made in fixing faults, of 7 open source software projects implemented in Java. Based on their experience, they proposed a fault classification scheme with relative frequencies of each fault type. As always, readers may…

Fuzzing the out-of-memory killer on embedded Linux

Kwan Yong Sim, Fei‐Ching Kuo, Robert Merkel

Software · Swinburne Research Bank (Swinburne University of Technology) · · cited 6× · DOI ↗ · more like this ·

Fuzzing is an automated black-box testing technique conducted with a destructive aim to crash (that is, to reveal failures in) the software under test. In this paper, we propose an adaptive random approach to fuzz the Out-Of-Memory (OOM) Killer on an embedded Linux distribution. The fuzzing process has revealed OOM Killer failures that cause the Linux kernel to remain in the OOM condition and become non-responsive. We have also found that the OOM Killer failures are more likely to occur when the Linux kernel has a higher over-commitment of memory requests. Finally, we have shown that the proposed adaptive random approach for fuzzing can reveal an OOM Killer failure with significantly fewer test inputs compared to the pure random approach.

TESTING A SOFTWARE-BASED PID CONTROLLER USING METAMORPHIC TESTING

Tsong Yueh Chen, Fei‐Ching Kuo, Wing K. Tam +1 more

Software · · cited 6× · DOI ↗ · more like this ·

The Proportional-Integral-Derivative (PID) controller is ubiquitous in industrial and military systems. Almost all PID controllers are now implemented as software in a microcontroller. Control systems are required to have very high reliability, particularly as they are regularly used in safety-critical systems. An effective testing technique is essential to achieve reliable PID controller software. Unlike simple control algorithms, PID controllers are capable of manipulating the process inputs based on the history and rate of change of the signal. It is very difficult to know whether the computation of the software is correct from the computed outputs. Previous research in other areas has shown that metamorphic testing is an effective technique for this kind of problem. In this paper, we examine metamorphic testing in the context of testing an embedded software PID controller based on…

Covering Array Constructors: An Experimental Analysis of Their Interaction Coverage and Fault Detection

Rubing Huang, Haibo Chen, Yunan Zhou +6 more

Software · The Computer Journal · · cited 5× · DOI ↗ · more like this ·

Abstract Combinatorial interaction testing (CIT) aims at constructing a covering array (CA) of all value combinations at a specific interaction strength, to detect faults that are caused by the interaction of parameters. CIT has been widely used in different applications, with many algorithms and tools having been proposed to support CA construction. To date, however, there appears to have been no studies comparing different CA constructors when only some of the CA test cases are executed. In this paper, we present an investigation of five popular CA constructors: ACTS, Jenny, PICT, CASA and TCA. We conducted empirical studies examining the five programs, focusing on interaction coverage and fault detection. The experimental results show that when there is no preference or special justification for using other CA constructors, then Jenny is recommended—because it achieves better…

Test Reconfiguration for Service Oriented Applications

Mark B. Cooray, J. H. Hamlyn-Haris, Robert Merkel

Computer Networks and Communications · Swinburne Research Bank (Swinburne University of Technology) · · cited 5× · DOI ↗ · more like this ·

Software testing in a service oriented and/or cloud computing environment is made challenging by its dynamic and loosely coupled nature. In this work, we propose and demonstrate a methodology for capturing changes and reconfiguring test data for an atomic service based on changes in the WSDL service description. Based on our results we believe that our technique can be extended further to capture changes in composite services and services that run on the Software as a Service (SaaS) platform in cloud computing.

An empirical study to review and revise job responsibilities of software testers

Tanjila Kanij, Robert Merkel, John Grundy

Computer Networks and Communications · Deakin Research Online (Deakin University) · · cited 4× · DOI ↗ · more like this ·

The broad domain of software testing includes different job responsibilities such as creating test plans, devising and running a variety of tests, documenting results, to liaising between different development teams. In this paper, we attempt to collate a list of software testing job responsibilities by applying three different social research methodologies to collect information from different sources. We found that “test” specific responsibilities are divided into several unit tasks including test suite generation, execution of test plans, and so on. We also found that along with test specific responsibilities, software testers must perform a number of other tasks common to other IT professionals in order to carry out their roles.