When Conformiq Designer 1.0 was released at the beginning of 2007 it had support for both test script generation as well as for online testing. The idea of online testing is simply that a model-based testing tool interacts directly with the system under test, instead of the tool generating a test script that can be executed later and independently of the MBT tool. Even though we supported both test execution modes, we dropped support for online testing in late 2008. Here the main reasons why.
Online Test Adapters are Hard to Create
Conformiq Designer was connected to systems under test in the online test execution mode through asynchronous online test adapters. These were software components that bridged together the model (or the online test behaviors generated from it) and the real system. In online test execution mode both sides (tester and system under test) could send asynchronously messages to the other side, and in addition to that the user could asynchronously pause and restart testing and change the test generation parameters. As a result the online components were notoriously hard to create even for seasoned software engineers.
Online Testing from System Models does not Scale to Fast Systems
Online testing from complex system models (such as those supported by Conformiq Designer) is algorithmically hard and therefore online test planning tended to consume lots of CPU power. This became a problem with fast systems that had short timeouts, e.g. micro controllers with millisecond scale timers. The timers would fire before Conformiq Designer had finished its planning cycle for the next test step, and tests then reduced to just series of fired timeouts. This is a fundamental issue with online testing, i.e. the fact that test planning has to occur in real time for timed systems.
People did not want it
People also did not want online testing when they were given the alternative of producing independent test scripts, because independent test scripts can be stored in version control and executed by people who do not have the skills to use an MBT tool, as well as by teams who do not have a license to the tool. Scripts can also (and do, in practice) leverage existing testing frameworks extensively, and because most test automation frameworks are script-based, the script generation approach allows for much more reuse than the online approach.
Online Testing Solved a Minor Problem
Today, the main reason to use online testing instead of off-line test script generation is to allow for testing of nondeterministic systems, that is, systems that can behave in multiple different, correct ways given the same series of inputs. On the surface it may sound that it is a sever limitation of the script-based approach that it does not easily allow for testing of heavily nondeterministic systems, but as a matter of fact the more nondeterministic a system is the harder it is to test in the first place. It is a good design practice to design systems for testability, and one important aspect of testability is that you can reproduce test results many times, and this implies that the system should be as deterministic as possible. The issue of testing nondeterministic systems is therefore, albeit real, relatively minor at least in the current practice.
Post Scriptum: So What About SpecExplorer?
Microsoft’s research tool SpecExplorer has an online test execution mode, so how does that reflect against what I wrote above? First, SpecExplorer does not run online testing using symbolic execution but through techniques known as slicing and explicit state enumeration. These make it easier to implement fast online testing, but make the approach inherently not scalable because human operators need to program in the slices in the first place, and if the slices are “wrong”, you don’t get the tests you need. Second, I think that the application area for online testing with SpecExplorer at least inside Microsoft has been narrower than the general market for Conformiq Designer, and this might have made it easier to come up with the necessary adaptation layers. Third, online testing is cool and it was and will be fun to implement it. So it fits well into a research tool. The practicality of full-fledged online testing in the industry with today’s technology still remains a question mark, however.