Metamorphic Testing And Metamorphic Relations: Definition And Examples

In this assignment, you are required to provide 6 distinct metamorphic relations for each of examples.

• A program is said to be testable if the output of any input can be verified

• To compute 41** (1/7)

Suppose the computed output is 1.7

How can we know whether this output is correct or not?

• To solve the following system of linear equations

3x + 2y – z = 4

x – 2y – 2z = -9

2x + y + z = 7

Suppose the solutions x=1, y=2 and z=3

• To find the values of x such that

x**67 + 3*(x**46) – x**37 +4.5 =0

Suppose the solutions for x are: 2.17, 6.5, ..

• A program is said to be non-testable if the output of any input cannot be verified

• A weather forecasting system which reports the amount of rain for a specific date

• A clinical x-ray system

• Compute the average for 10 million real numbers

• Compute sine function

• sin function

  – sin(0o )=0

  – sin(30o)=0.5

  • Suppose the program returns:

  sin(29.8o )=0.49876 correct?

Shortest path program SP(G, a, b)

where G is a graph, a is the starting node and

b is the destination node

SP(G, a, b) returns a path like:

a – x – y - …. - s – t – b

Find all possible paths from node a to node b

Check against all these possible path to see

whether SP(G, a, b): a – x – y - …. - s – t – b is the shortest

• Absence of test oracle
• Test oracles are available but too expensive to be applied

To search how many times and where a string

(S1) appears in another string (S2)

S1 is abaccabacddaabcabdcdaabacccdbabaddc

S2 is abac

abaccabacddaabcabdcdaabacccdbabaddc

Various approaches

• Machine learning (Kanewala and Bieman)
• Data mutation (H. Zhu)
• Coding (Zhang Chen Hao Xiong…)
• Composition (Liu et al.)
• Category-choice framework (Chen et al.)

What is Metamorphic Testing?

This report ensures to find the metamorphic relation. Metamorphic testing (MT) is a technique of property-based software testing. This method of testing is highly effective to address the issues in oracle and test case generation issues. It is also noted that, determining the desired results from the chosen test cases is highly complicated and hard. It can either be stated as, determining whether the actual outputs agree with the expected outcomes is difficult. The problems of oracle denotes the circumstances where it is to a great degree troublesome, or incomprehensible, to confirm the test consequence of a given experiment (that is, an information chose to test the program). Ordinarily, after the execution of an test case t, an efficient component called a test oracle (or just an oracle) is required to check the execution result. In the event that, the outcome does not concur with the normal result, we say that t comes up short and results as a disappointment and contains issues in generating the test cases. Else, we say that t succeeds and its result is fruitful, or non-disappointment causing, test case. In some genuine circumstances, an oracle may not exist, or it might exist but the requirements are limited which make it infeasible to utilize.

This report contains six distinct metamorphic relations, for each of the examples provided  in the lecture notes 3, form the slide number 37, 38, 39, 40 and 41 will be discussed and solution will be presented for Metamorphic Testing. Thus, the report aims to find the metamorphic relation.

The objective of this report is to provide definition for every single metamorphic relations along with example. 

Initially, during 1998 the metamorphic testing was introduced in a technical report, but the very initial major survey in the MT field was carried out during 2016. The Metamorphic testing (MT) denotes a method to produce follow-up test cases in the light of present test cases which have not revealed any disappointment. The Metamorphic testing must o be connected related to other test case determination techniques which actually creates the underlying arrangement of test cases. For example, consider a program p implementing function f on the domain denoted as D. Assume S as the test case selection strategy which the analyzer has adopted like the data flow testing or branch coverage. As indicated by S, a test set T = {t1, t2, ..., tn} ⊂ D, where n ≥ 1, can be produced. Running the program on T yields the yields p(t1), p(t2), ..., p(tn). At this point, when there is oracle, these test outcomes can be confirmed against f(t1), f(t2), ..., f(tn). The metamorphic testing is valuable in light of the fact that by far most of test cases are effective, despite the fact that they have not uncovered any disappointment, these test cases do convey helpful data disregarded in customary testing. The metamorphic testing creates a follow-up of test cases by making reference to metamorphic relations (MRs).

Metamorphic Relations

Since the metamorphic testing evaluates the relations among a few executions as opposed to the accuracy of individual yields, the metamorphic testing don’t require oracle, as it could be automated completely. Such a methodology is additionally connected to fault-based testing without oracles. It can examine where a trial MT system is developed (Zhou et al., 2004). The metamorphic relations are not limited just to the programs which have numerical inputs or equality relations.

The problems of oracle refers the circumstances where it is to a great degree troublesome, or incomprehensible, to confirm the test consequence of a given experiment. Once the execution of a test case t is completed, an efficient component called a test oracle is needed to check the results of execution (Chen and Kuo, 2018). Therefore, the metamorphic testing aims to alleviate the issues related to oracle. It is basically a property-based testing method, which provides a new point of view for designing the test cases. Additionally, the metamorphic testing checks for the metamorphic relation (Yueh Chen, 2018).

The disadvantages of metamorphic testing are listed below:

• It only provides limited Output Information.
• It is not able to identify Certain Faults.
• It contains issues in implementation.

A program is referred as the testable program in case it is possible to verify the output of any input.

To compute 41 ** (1/7)

=> 0.836

If Answer is 1.7

=> 41 ** 24

=> 1.7

In this section the following system of linear equations are solved:

 3x + 2y – z = 4,

 x – 2y – 2z = -9 and

2x + y + z = 7, where the solutions x=1, y=2 and z=3.

Solution:

Apply the value of x, y and z, in the provided equation.

1. 3x+2y-z=4

3+4-3 = 4

     4= 4

1. 1 - 2x2 – 2x3 = -9

1 -4 -6= -9

• 2x1 +2+3 = 7

2+2+3= 7

    7=7

Therefore, the given solution is correct.

Here, the values of x such that x**67 + 3*(x**46) – x**37 +4.5 =0, where the solutions for x includes either of the following: 2.17 and 6.5.

Solution:

n = 2.17

n ** 67 + 8* (n**46) – n **37 + 45 = 0

If n = 2.17

2.17 ** 67 + 8* (2.17**46) – 2.17 **37 + 45 = 0

=3.48

If n = 6.5

6.5** 67 + 8* (6.5**46) – 6.5 **37 + 45 = 0

=2.9

Therefore, the given solution is incorrect.

A program is referred as the non-testable program in case it is not possible to verify the output of any input.

In terms of oracle, there will be challenging problems. For example, in machine learning the Applications, simulation, scientific computing, optimization, and so are at times non-testable programs. The reason for this is that, the expected outputs are difficult to be identified or the outputs actually doesn’t exist at all. Therefore, the computer scientists have tried their best to find various methods which can address the problems related to oracle, and this is available during the software verification process (Liu, 2009).

Examples of Metamorphic Relations

The sin function used are,

-sin (0⁰) = 0

-sin (30⁰) = 0.5

Where, whether the program returns sin (29.8⁰) = 0.4987 or not is verified in this example.

Solution:

-sin (x) = sin (x+360)

Compute 29.8⁰ + 360⁰ =389.8⁰

sin (29.8⁰) = sin (389.8⁰)

29.8⁰ + 360⁰  =  398.8⁰

0.4987+0 = 0.4987

Solution:

The sin function used are,

-sin (0⁰) = 0

-sin (30⁰) = 0.5

Where, whether the program returns sin (29.8⁰) = 0.4987 or not is verified in this example.

Solution:

Therefore, sin (29.8⁰) = 0.4969 is correct.

Test Oracle is a mechanism or a procedure against which the computed outputs can be verified.

Here, by comparing the Example 2 with the Example 3, the outcomes of Example 2 can be verified, which will represent the metamorphic relations.

Here, it is identified the number of times the string is searched, where the string (S1) appears in another string (S2).

S₁ =abaccabacddaabcabdcdaabacccdbacaddc

S₂= abac

abaccabacddaabcabdcdabacccdbabaddc

Solution:

The correct out should be (2, 6, 21), because there are two occurrences. The first occurrence starts at the position 6 of S₂ and the second occurrence starts at the position 21 of S_2.

This section provides details on whether is it feasible to identify or generate the metamorphic relations (MRs). The answer to this includes that, among various approaches the machine learning is identified.

The metamorphic testing refers to a method to automate the testing process for the programs without test oracles.

Conclusion

In this report, the 6 distinct metamorphic relations for each of the examples provided in the lecture notes 3, form the slide number 37, 38, 39, 40 and 41 are discussed and solution are presented for Metamorphic Testing. Moreover, for every single metamorphic relations, apart from its definition, an example must is also provided.

The definition of metamorphic testing, testable program and non-testable program is provided. It is observed that metamorphic testing refers to a technique of property-based software testing. This method of testing is highly effective to address the issues in oracle and test case generation issues. It is also noted that, determining the desired results from the chosen test cases is highly complicated and hard. It can either be stated as, determining whether the actual outputs agree with the expected outcomes is difficult. This report finally determines the metamorphic relations, where its novelty and elegance is represented. Example 1 and example 2 are compared to retrieve the desired outcomes. It is observed that the testable program is a program whose input’s output can be verified. The Example 2 of Testable Programs is verified as correct, whereas the Example 3 of Testable Programs is verified as incorrect.

References

Chen, T. and Kuo, F. (2018). Metamorphic Testing: A Review of Challenges and Opportunities. ACM computing surveys, 51(1).

Liu, J. (2009). Metamorphic Testing and its Application on Hardware Fault-Tolerance. Dept. of Electrical and Computer Engineering University of Wisconsin – Madison.

Yueh Chen, T. (2018). Metamorphic Testing. Swinburne University of Technology. [online] Available at: https://pdfs.semanticscholar.org/presentation/4544/bd04340fb5052c66c944dd5432e36f2ff6ae.pdf [Accessed 21 Sep. 2018].

Zhou, Z., Huan, D., Tse, T., Yang, Z., Huang, H. and Chen, T. (2004). Metamorphic Testing and Its Applications. Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004).