The stuff that I often write about pertains to performance of test generation. The reason for this is quite clear: performance has been and still is one of the biggest stumbling blocks in deploying model based testing and automated test design in an industrial context. We, as a technology provider, are not the only one affected by this. In fact the scalability issue with complex real-world models occurs in all the tools that use variants of the state-space exploration paradigm*** to process the models, for example Conformiq Creator and Designer.
The question is what your tool vendor is doing to alleviate the issue and push the performance envelope. Some tools try to ease the state-space explosion problem by pushing the problem to the user (e.g. allowing the user to “slice” or “restrict” the model using “use cases”), whereas some try to provide a more technological solution. We fall to this latter category, as showcased for example by our unique distributed test generation back-end (the Conformiq Grid solution). And why go the technology route? Because pushing the problem to the human operators starts to eat the productivity gains.
Nowadays the trend with some of our customers has been on building extremely big models which unfortunately have been almost beyond the capabilities offered by our tooling. That’s why, over the past 12 or so months, our R&D has been heavily focusing on improving the test generation performance of our Conformiq Creator platform and the results of these efforts have been incorporated into Creator 4.0 that has just been released. And to be honest, we are quite excited about this release!
Firstly the model loading (model compilation, extraction of test targets from the model, and so on) is almost instantaneous. This is a significant improvement over previous versions of the technology where model loading times were up to several minutes and the process consumed a significant chunk of your computer memory. However, even bigger performance improvement stems from the updates we have implemented on the state-space exploration algorithm and on the way that we internally preprocess and analyze the Creator models. It wasn’t uncommon in the past that test generation for very large complex models would take several hours. Not so anymore with Creator 4.0! According to our benchmarks Creator 4.0 is able to finalize the state-space exploration and to generate tests in a matter of minutes even from those huge models. While that may still seem like a lot of time,understand that the amount of calculations that our technology needs to perform in order to deliver the level of test design automation that it does is astronomically large.
That’s actually one of the main arguments to use an advanced technological solution such as Conformiq Creator and Designer. The end user, that is you and me, does not need to bother with all the insanely complex combinatorics and all. Instead we offload the hard work to an algorithm, which with Creator 4.0 is faster than ever before, improving our productivity and quality of work. If the problem is difficult for a computer to solve, you can bet that it’s extremely difficult for us humans to do so as well. And while the algorithm is crunching those numbers and finding tests that are otherwise almost impossible to come by using a manual thinking process, maybe you can check out all the other cool stuff that Creator 4.0 introduces!
*** It is worth pointing out that the vast majority of the test generation tooling out there are relying on relatively simple, even trivial, technical solutions demonstrated, for example, by the fact that virtually all of them separate design and data by pushing the test data design to the end user. Just selecting the data used to design test cases can take 40-60% of the overall test design time and Conformiq tools design the data together with test cases automatically.