Economics of Software Testing Using Discrete Approach

1. Introduction

Computer systems are playing a vital role in our daily life. It has become integral part of work environment and the computer systems with high reliability is the need of hour. Reliability of the computer systems is highly dependent on the quality of software installed into it. During software development human errors may result in malfunctioning of the system. This is why a skilled testing team is required for developing a software system to build high quality software. This can be achieved through continuous software testing as it helps in removal faults causing failure of the software system. To help the testing team in planning the testing phase, academicians and researchers developed mathematical models to estimate the count of errors in the system and named it as software reliability growth models (SRGM). These SRGM are used to estimate the reliability growth of the software system and the count of residual errors at a particular point of time. Two research directions were adopted by the researcher to model SRGM i.e. calendar time and test case execution based SRGMs. The first SRGM was developed by Goel and Okumoto (1979) considering that the fault detection rate is constant. This SRGM follows exponential distribution hence named as exponential SRGM. Also, the unit of software fault detection period used by this model was machine execution time or calendar time. However, at times the number of executed test cases or the number of test runs are considered appropriate unit for fault detection period (Brooks and Motley,1980). Later Yamada et al. (1983) developed SRGM by stating that fault detection and correction is a two-stage process i.e. fault detection and fault correction. Their SRGM followed S-shaped distribution and hence named as S-shaped SRGM. Yamada and Osaki (1985) proposed a discrete SRGM based on non-homogeneous Poisson process (NHPP) where the random variable is denoted as the number of faults detected by ‘n’ number of test case runs. Kapur and Garg (1992) developed a flexible SRGM considering logistic distribution function. Researchers further developed calendar time and test case run based mathematical models by combining the cost incurred in testing and operational phase to solve another important problem of determining duration of the testing process (Okumoto and Goel,1980; Kapur et al.,1994). The importance of determining the optimal testing duration is due to the trade-off between the benefit and loss of early and late release. In case the testing process is stopped too early, user will face many failures while operating due to presence of many faults in the software. This in turn would cause significant losses owing to failure penalty and goodwill loss due to dissatisfied customer. On the other hand, delaying the software release to bring a more reliable software in the market may increase the software development cost too much and the firm may suffer market opportunity loss as well. Various optimization problems were developed to estimate the optimal time of release of the software under the cost, reliability and failure intensity constrain set by the management (Kapur, Garg & Kumar,1999). Okumoto and Goel(1980) formulated two types of release time policies using exponential SRGM under unconstrained cost and reliability objective. Kapur et al. (1994) developed test run based optimal release policies considering the impact of imperfect debugging. Later on different factors and improvements were incorporated to enhance the accuracy of the output. (Majety et al., 1999; Quigley, 2005). Many researchers and academicians developed SRGMs with varied assumptions and applied them to obtain the optimal release time (Kapur, Garg & Kumar, 1999; Pham, 2006). Jiang et al. (2003) started the discussion of differentiating between the time of releasing a software and the time to stop testing. Arora et al. (2006) proposed another research direction in determining the software release time. They suggested a mathematical model to show the benefit of early release. They supported the theory of early release and then continuing with the process of testing the software in the operational phase. Before this research, it was assumed that the time of release of a software marks the end of software testing phase as well (Figure 1). Arora et al.(2006) claimed that the total software development cost is minimum if release and testing termination times are separated (Figure 2).