How Effectively Does Metamorphic Testing Alleviate The Oracle Problem?

How Effectively Does Metamorphic Testing Alleviate the Oracle Problem.

Primary context of the paper is to emphasize over the Metamorphic testing efficiency and effectiveness for reducing the “Oracle problem”. In this critical reviewing, the article is identified as “How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?” by “Huai Liu, Fei-Ching Kuo, Dave Towey, and Tsong Yueh Chen”. Oracle is identified as something that can vary the test case appropriateness results. In case, the Oracle too expensive for using or does not exist, and then Oracle problem occurs. Now, the author’s purpose is to depict the metamorphic testing as an approach of helping appropriateness of a program. The purpose is justified for experienced testers with smaller amount of testing required. The author needs to cover the testing expertise for writing the article. The thesis statement is to study metamorphic testing for redacting Oracle problem. The researcher agrees with the thesis statement as it demonstrates the appropriate way to reduce the problem.  

Summary

Metamorphic testing is identified as the testing procedures with various applications in certain domains as successfully recognizing the faults in programs. There were some studies to establish suitable guidelines in test Oracles. The study is conducted to identify suitable prior studies and related works. Therefore, the study is carried forward with secondary approach following article reviewing, journal studying, and relevant work analyzing. The prior studies show that Oracle problem can be taken into consideration with formal algorithm procedures as finite state machines. Compared to other methods, metamorphic testing is generic approach. Moreover, the assertion technique is embedded within source code for detecting bugs while execution. On the other hand, multiple versions of program can be utilized for reducing the Oracle problem. The technique is identified as N-version programming as it is designed to be fault tolerance technique. Mutation analysis technique is suitable sometimes with constructing the test Oracles. It is more of a unit testing along with list of assertions. Hence, from the secondary studies, the research questions are identified as following:

RQ1: How effectively can metamorphic relations detect software faults?

RQ2: How capable are testers of alleviating the oracle problem using metamorphic testing?

RQ3: How the cost-effectiveness of metamorphic testing can be optimized?

The study was carried out to depict the metamorphic testing relation effectiveness for detecting software faults and bugs. Moreover, the study is elementary for identifying the capability of testers for reducing the Oracle problem along with metamorphic testing. Furthermore, metamorphic testing can be cost-effective or not is another research aim to be achieved.

Critique

The author is proposing the research questions in form of hypotheses as with positive or negative considerations. The positive and negative consideration is proposed for making the study empirical along with meeting the objectives. Author is proposing that metamorphic testing have fault-detection effectiveness for reducing the Oracle problem. The author will assess the metamorphic testing with qualitative approach for identifying how many metamorphic relations are required to reduce the problem. Again, the in-depth analysis of metamorphic testing is performed for assessing metamorphic testing relations, overall performance, and tester capability evaluation. For the thirds research question, the study identified some important factors as a way to understand possibility to optimize the cost and effectiveness of utilizing metamorphic test procedures. Now, the study is mostly qualitative, analytical, and secondary study based. On the other hand, there is an experiment conducted for subjected programs, mutant generation, metamorphic testing relations recognition, random testing whether with or without that particular problem and test case preparation.

The experimental results are identified as detection of real-time faults as in MultipleKnapsack and Sparse-MatrixMultiply programs. However, the testers have recognized the faults with metamorphic relations. Again, the prior studies of metamorphic testing shows that popular Siemens suite have such real-time faults as well. Now, for first RQ, the hypothesis significance is viewed as 0.05 and p-value is lower than significance value, resulting in rejection of null hypothesis (H₀). As per the results, there were 19 metamorphic relations and each of them has outnumbered the RT with 90% of difference in between the RT and RT_o. Now, (6 + 10 + 8 + 12 + 19) / 99) that is 55.56% have fault-detection values between RT and RT_o. In case of using all metamorphic test relations, the collection was identified with another way to regard them for imitation of test Oracle. The fact is identified as a smaller amount of diverse metamorphic relation is adequate to match the fault-detection showing intuitively appealing results. A team of 4 to 7 testers can easily identify the metamorphic relations for similar detection of faults to test oracle. Moreover, the discussion demonstrates the insights of metamorphic testing for best practice of conducting such testing. 

The primary argument is identified in terms of setting the hypotheses in the study. The insights are identified in the study with showing how the testing can be properly conducted with best practice adoption. The diversity of metamorphic study can be recognized with important than the quantity. Moreover, smaller team of testers with separate expertise can be effective than larger teams in testing procedures. The researchers have recommended incorporating the testers for diversity consideration for metamorphic test relation selection for testing. The insight presented in the article is strong and it is not open for falsification as the result is justified with significance level of hypotheses.

Key words

In the article the key words are identified as following: metamorphic relation, software testing, oracle problem, test oracle, and metamorphic testing. Software testing is properly addressed within the study with recognizing the metamorphic testing as a suitable approach. Test oracle is main context in the study as it is depicted with metamorphic testing relations and procedures. Moreover, the oracle is identified as something that can vary the test case appropriateness results. In case, the Oracle too expensive for using or does not exist, and then Oracle problem occurs. Furthermore, the key words are identified in the article following individual evidences of such terminologies from prior studies as well.    

The author’s are Huai Liu, Fei-Ching Kuo, Dave Towey, and Tsong Yueh Chen. Huai Liu is with Australia-India Centre for Automation Software Engineering, RMIT University, Melbourne 3001 VIC, Australia. Fei-Ching Kuo and Tsong Yueh Chen are with the Faculty of Information and Communication Technologies, Swinburne University of Technology, Hawthorn 3122 VIC, Australia. Dave Towey is with the Division of Computer Science, The University of Nottingham, Ningbo, 315100, Zhejiang, China. They have properly established their expertise in preparing the study with metamorphic test relations and related test case generation. Moreover, the author’s have selected this particular argument of metamorphic testing to show it cost-effectiveness, fault-detection technique, and overall tester capability.

The evidences are identified in the inside and other investigation of transformative testing is performed for surveying changeable testing relations, general execution, and analyzer ability assessment. For the thirds look into question, the study distinguished some critical components as an approach to comprehend plausibility to enhance the cost-adequacy of changeable testing. Presently, the study is for the most part subjective, explanatory, and optional study based. Then again, there is an examination led for subjected programs, mutant era, changeable testing relations acknowledgment, arbitrary testing whether with or without that specific issue and experiment readiness. The test results are distinguished as location of continuous blames as in some JAVA programs. Notwithstanding, the analyzers have perceived the shortcomings with changeable relations. Once more, the earlier investigations of transformative testing demonstrates that mainstream Siemens suite have such continuous blames also. Presently, for first RQ, the theory importance is seen as 0.05 and p-esteem is lower than criticalness esteem, bringing about dismissal of invalid speculation (H0). According to the outcomes, there were 19 changeable relations and each of them has dwarfed the RT with 90% of distinction in the middle of the RT and RTo. Presently, 55.56% have blame discovery adequacy amongst RT and RTo. In the event of utilizing every single transformative connection, the accumulation was related to another approach to respect them for impersonation of test Oracle. The truth of the matter is recognized as a littler measure of different transformative connection is satisfactory to coordinate the blame location indicating instinctively engaging results. A group of 4 to 7 analyzers can without much of a stretch recognize the transformative relations for comparative discovery of deficiencies to test prophet. Additionally, the exchange exhibits the bits of knowledge of changeable testing for best routine of leading such testing.

Conclusion

The context relies upon the software testing as well as the Metamorphic Testing for mitigating the impact of Oracle problem. As metamorphic testing is an effective way to test any source code or program with metamorphic relations and testing procedures. The article has offered empirical analysis of the metamorphic testing for reducing the impact of Oracle problem where the testers can detect the problems successfully. Several postgraduate and undergraduate students were taken in the study for quantitative research and identification of knowledge of metamorphic testing. Again, the fault-detection effectiveness is studied with group of testers to link the number of metamorphic test relations in subjected programs. More importantly, the smaller diverse metamorphic relations can identify the most underlying situation of Oracle test. The Oracle testing is based on critical factors as having impact on metamorphic relations. The diverse metamorphic and metamorphic test relations may have fault and error detection efficiency for programs and as well within the study.

Chen, T.Y., Kuo, F.C., Towey, D. and Zhou, Z.Q., 2012, August. Metamorphic testing: Applications and integration with other methods: Tutorial synopsis. In 2012 12th International Conference on Quality Software(pp. 285-288). IEEE.

Hui, Z., Huang, S., Ren, Z. and Yao, Y., 2013. Metamorphic testing integer overflow faults of mission critical program: A case study. Mathematical Problems in Engineering, 2013.

Kanewala, U., 2014, March. Techniques for automatic detection of metamorphic relations. In Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on (pp. 237-238). IEEE.

Liu, H., Kuo, F.C., Towey, D. and Chen, T.Y., 2014. How effectively does metamorphic testing alleviate the oracle problem?. IEEE Transactions on Software Engineering, 40(1), pp.4-22.

Pullum, L.L. and Ozmen, O., 2012, December. Early results from metamorphic testing of epidemiological models. In BioMedical Computing (BioMedCom), 2012 ASE/IEEE International Conference On (pp. 62-67). IEEE.

Ramanathan, A., Steed, C.A. and Pullum, L.L., 2012, December. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics. In BioMedical Computing (BioMedCom), 2012 ASE/IEEE International Conference on (pp. 68-73). IEEE.

Segura, S., Durán, A., Sánchez, A.B., Berre, D.L., Lonca, E. and Ruiz?Cortés, A., 2015. Automated metamorphic testing of variability analysis tools. Software Testing, Verification and Reliability, 25(2), pp.138-163.

Sim, K.Y., Wong, D.M.L. and Hii, T.Y., 2013. Evaluating the Effectiveness of Metamorphic Testing onEdge Detection Programs. International Journal of Innovation, Management and Technology, 4(1), p.6.