Some companies implement parts of their products using Model Driven Engineering (MDE). In this approach, implementation code is generated, at least partially, from system models. In those domains where it works, the MDE approach can lead to significant cost savings and quality improvement. It is then natural to ask if the same implementation models can be reused for automatic test generation. But is it a good approach?
There is one, and only one, big problem in reusing implementation models for automatic test generation, and that is the loss of independent testing. If you generate your system’s code from a model, and then use the same model as a reference for the correct operation of the system, you are not going to find any functional defects. The only thing your automatically generated tests verify is that the system works as it works, because you are using the system itself as the reference for its correct operation.
However, tests generated from an implementation model can be very useful for the following purposes:
- Regression testing
- Testing the generated code in its deployed context (including hardware, operating system, hand-written drivers, integrated components, and so on)
- Driving up white-box coverage metrics, especially when those are used as gating criteria in the release process
Here’s a concrete example. You are implementing a simple calculator application and your model accidentally states that the correct way to calculate the sum of two numbers x and y is to execute the action “sum = x – y”. The generated code then calculates sums incorrectly using subtraction. When you feed now the same model to a system model driven test generator, you are likely to get a test case like the following:
- Enter 5
- Enter 8
- Press +
- Expect result -3
…as this is behavior that is dictated by the model to be correct. When you execute this test case against the implementation the test will pass. The problem is that the same functional defect that is in the actual implementation is used as a reference to judge its correctness.
This is why we recommend that testing models should be independent from the implementations, at least when it comes to behavior. Sharing data structure and class declarations does not create the same problem and indeed reusing static structure model parts can be a great way to reduce modeling and model maintenance burden.