Model Driven Testing

Conformiq Qtronic vs. Conformiq Test Generator

July 17th, 2007 by admin

As you can see from our web pages our product offering currently consists of two products: Qtronic and Conformiq Test Generator (aka CTG).

I’m often asked about the difference between the two. This is especially the case with those who have used CTG in the past and now try to bend their minds around Qtronic “modus operandi”.

Conformiq Test Generator represents our 2nd generation test generation technology and was released in autumn 2002 (this was preceded by Swiftest, which we consider the 1st generation). Qtronic with roughly six months in the market, on the other hand, represents the 3rd generation “state-of-the-art” model-driven test automation. At this point it should be made clear that Qtronic as we know it today is not a successor to CTG in the sense that as a CTG-user you could just take your existing CTG test model and migrate to Qtronic.

The fundamental difference between Qtronic and CTG is in what the model used as the basis for test generation describes.

With CTG you define a test model, which is essentially a description of the environment of the System Under Test. If you were testing a server your test models would define the functionality of the client that interacts with the server. In the end this boils down to UML-based graphical test scripting.

Qtronic, on the other hand, takes in the design model/behavioral description of the SUT itself, thus implementing true model-driven test generation. Returning to our client-server example with Qtronic you would define the behavior of the server itself in order to test the server.

In a way you could see these two models as mirror images of each other; where a design model has an input, a test model has an output and vice versa (this is a simplification because in the case of non-deterministic models such simple “inversion” doesn’t work). The virtue of Qtronic is that it carries out this (mathematically challenging) “inversion” of a system design into test cases, which with CTG, given the same system design spec, has to be done manually by the test model designer. Qtronic hence eliminates the need for test design.

From a process point of view you could say that CTG is a more traditional testing tool whereas Qtronic carries through the different phases of a software process by providing support for automatic deployment of early-stage design artefacts not only in development but also in testing. And well-defined designs specs of any system make for excellent documentation of the said system as well.

Posted in Uncategorized | Comments Off

ICST Call for Contributions

July 4th, 2007 by Antti Huima

An new series of IEEE international conferences launches next April in Norway on the subject area of “software testing, verification and validation”. I post the call for contributions on our Model Driven Testing Blog because model-based testing is one of the key areas for this new conference. The call is below:

CALL FOR CONTRIBUTIONS

1st IEEE International Conference on Software Testing, Verification and Validation (ICST 2008)

Sponsor: IEEE Computer Society Technical Committee on Software Engineering (TCSE)

April 9-11, 2008, Lillehammer, Norway

http://www.cs.colostate.edu/icst2008/index.html

There has been very strong activity in the field of testing and validation research, as reflected in recent years by numerous workshops dealing with testing in different areas. The new IEEE International Conference on Software Testing Verification and Validation (ICST) will offer an open forum for software testing, verification and validation research and its transfer to practice. Among other things, it aims at stimulating scientific research on model-based software testing, domain specific testing, empirical studies of testing techniques, and the technology transfer of research results to software development practices.

We invite original, high-quality research papers and industrial experience papers in all areas of software testing, verification and validation.

Topics of interest include, but are not limited to:

- Software testing theory and practice
- Model-based testing
- Domain specific testing including, but not limited to,
security testing, web services testing, database testing,
and OO software testing
- Verification & Validation
- Quality Assurance
- Model checking
- Empirical studies
- Metrics
- Inspections
- Tools
- Testability and diagnosability
- Design for testability
- Testing education
- Testing in multidisciplinary applications
- Embedded software
- Technology transfer
- Model-Driven Engineering
- Novel approaches to software reliability assessment

Conference Chair: Lionel Briand
Program Chairs: Rob Hierons, Aditya Mathur
Workshop Chair: Benoit Baudry
Industry Chair: Per Runeson, Clay Williams

Important dates:
Submission of abstracts: October 5, 2007
Submission of full papers: October 12, 2007
Notification: January 4, 2008
Camera-ready: February 1, 2008

For ICST 2008 submission guidelines and all additional up to date information, please visit http://www.cs.colostate.edu/icst2008/index.html

Posted in Uncategorized | No Comments »

Challenge to Our Competitors: Provide Free Evaluation Downloads

July 3rd, 2007 by Antti Huima

Conformiq Qtronic has been available on the web for completely free, even anonymous, evaluation download since the tool was commercially released. However, our customers have had big difficulties in getting the tools from our competitors for evaluation.

One said that “we are evaluating also the Leirios tool… even though they do not provide an evaluation [download]”. A Finnish researcher from the governmental research institution writes in his blog that people from T-Vec answered that “they do not provide evaluations for students and neither to Finland”.

Reactive Systems, however, provides an evaluation download.

As far as we know, the only tool that can claim to be on par with Conformiq Qtronic with regards to generation of test data is, however, the Leirios Test Designer… so we pose this concrete challenge to our competitors and especially the French company: make an evaluation copy of your tool available so that users of model-based testing can make cost-efficient and informed purchase decisions! The commercial model-based testing market is still in quite early stages, and people need to spend lots of effort in understanding the globally available offering… by providing free evaluation downloads you would contribute significantly to awareness about model-based testing and the maturity of tools.

There are many dimensions in which the model-based testing users need to be able to compare the tools, including:

The ease of adoption and use of the system modeling languages. There is much difference in the solutions; Qtronic uses C#/Java + UML, Leirios tool uses (as far as we know) UML + OCL, Reactis is based on Simulink and Stateflow…
The speed of test case generation. There is almost no information about how fast the tools are for generating test cases, even though it is known that computationally intensive algorithms are required.
The quality of generated test cases. Once the test case generation runs the important question is what you get out of it. This is very context-dependent so it is very important for the model-based testing users to experiment with test case generation in their own context.
Availability of online testing. Online testing provides benefits over offline test case generation when system models are nondeterministic. Nobody can neglect the fact that both the academic community as well as the commercial companies (in addition to Conformiq, for example Microsoft in their SpecExplorer tool) have recognized the importance of online testing.
User experience in general.

So please answer to our challenge; provide your tools freely available for evaluation for the better of the model-based testing users as well as for the advancement of the model-based testing market!

Posted in Uncategorized | No Comments »

Finnish Student Blogs About Qtronic

June 27th, 2007 by Antti Huima

We run the query for “Conformiq Qtronic” accidentally on the Finnish Google site and found an interesting blog entry in Finnish, where a guy who calls himself oopee writes about the tool… in Finnish, unfortunately. The Finnish post is linked here. Here are few comments on the post.

First, oopee deduces that a large part of the programming was outsourced but this is not actually true as all parts of Conformiq Qtronic have been developed in-house… save for the Trolltech Qt GUI toolkit of course which we are using happily since 2002.

Besides this, oopee mentions three main points why “he thinks that Qtronic is good”: open interfaces, good test generation algorithms, and convenient licensing options. It is great to hear this because we have indeed invested in (1) making the tool open and malleable and (2) having world’s best test generation algorithms!

The negative thing oopee says about Qtronic is that it is “terribly expensive”, but I guess that this becomes “less true” when you compare the tool’s price with (1) how much you pay for manual test case design and test case maintenance and (2) how much you pay for missing, incorrect, invalid and replicated test cases and the related validation and debugging efforts…

Posted in Uncategorized | No Comments »

TESTCOM/FATES, Tallinn, Estonia

June 27th, 2007 by Antti Huima

I arrived yesterday to Tallinn, Estonia by a catamaran from the south coast of Finland, a 90 minutes ride that was surprisingly smooth, to take part in the combined TESTCOM/FATES 2007 conference (co-located with FORTE).

Today morning I gave my invited talk on “Implementing Conformiq Qtronic”, in which I surveyed the basic architecture of our model driven testing tool and the different challenges we have met while working with the implementation.

I think that the two important messages I tried to put forth were that (1) it is possible to do successful model driven testing with infinite state space models and (2) that it requires much more than a test generation algorithm to make a commercial-quality model driven testing solution.

After my talk, the program has continued with the presentations of peer-reviewed articles and at least today the focus has been pretty much on finite state machines… with all (due) respect, I believe it is about the time for the research community to step out of this FSM sandbox and start to look around for the real world!

Posted in Conferences and Events | No Comments »

Model Driven Testing Tutorial at TTCN-3 User Conference

May 29th, 2007 by Antti Huima

I gave today a tutorial on “model driven generation of TTCN-3 test cases” at the TTCN-3 User Conference 2007. The audience at the Ericsson building in Kista, Stockholm was maybe from fifty to one hundred people.

In this blog entry I will go through some of the questions people asked from the audience.

Q: “You said that model driven testing eliminates the risk of incorrect test cases because test cases are generated by an algorithm instead of a human. But the model itself is designed by a human, so isn’t the risk there still?”

A: “Yes, the model is designed by a human being and there is no way around this. But the model is nearer to the functional requirements than the test cases and is a smaller artifact than the corresponding test case library in general. Therefore reviewing the model and ensuring its correctness takes less time and incurs less costs.”

Q: “Does your approach also help in generating the test data as opposed to test behavior?”

A: “In our approach there is really no clear separation between test data and test behavior. If your system’s behavior depends on the data it receives, then you must model the data, and once you do this you get test data also in the test cases. Our solution has full support for the generation of test data because without it you cannot realize the desired behavior in any form either.”

Q: “Is the TTCN-3 code [shown in demonstration] completely computer-generated? Also the comments?”

A: “Yes. Of course our product cannot comment intelligently on the meaning of the model, but for example if you include links to your system requirements in the model, these links are visible also as comments in the generated test cases.”

Q: “You did not mention state space explosion in your talk.”

A: “It is certainly true that you can create models that will choke up our solution, there is no doubt to that. But on the other hand the testing heuristics we use (e.g. boundary value analysis of condition coverage) scale up linearly or low-degree polynomially in the size of the model, instead of e.g. exponential growth. This means that the expected number of test cases is manageable. Also, because our tool uses internally symbolic methods for state space exploration, the state space explosion problem takes a different form in our context.”

For more information about TTCN-3, check out www.ttcn-3.org.

Posted in Conferences and Events | No Comments »

Toolapalooza 2007

April 19th, 2007 by Antti Huima

Hi there! This entry comes from the Cisco Toolapalooza 2007, Cisco’s by invitation trade show for testing tool vendors related to Cisco’s business.

It’s always striking how the people on the Silicon Valley (and U.S. in general) are open and direct and up-to-the-point when compared to their European colleagues. It’s definitely a different atmosphere—the sense of efficiency, if you wish, is everywhere in the air.

This event was shadowed, most unfortunately, by a horrendous massacre at the Virginia Tech, certainly the number one news in the States for many days in a row. The country is showing respect to the murdered and their families and the flags are half-way down everywhere.

Our presentation on our offering is still upcoming in a few hours, and there will be also a presentation on model-based testing by, it seems, Cisco people themselves. It makes sense as the guys here are always scanning for new ideas to make their life better and their products more competitive. Certainly many things could and should be learned by the European industry from the attitude here. No wonder that States is the #1 power in software industry.

Here at the end a picture of my colleague happy at the stand…

Posted in Conferences and Events | No Comments »

Greetings from Portugal

April 3rd, 2007 by Antti Huima

I spent the last weekend in sunny Braga, Northern Portugal. The occasion was the Model Based Testing 2007 workshop, held in conjunction with the ETAPS conference. The country was beautiful, weather pleasant and food very cheap (at least when compared with the Finnish price level).

On the scientific level, the quality of the papers was mixed. Maybe one of the highlights was the presentation by Swedish researchers about efficient test generation for timed systems. This reasearch was, of course, coupled with the well-known UPPAAL toolset and its derivatives.

On Sunday morning I gave my invited talk on the use of model-based testing in standardization. This joint presentation by Mr. Stephan Schulz from ETSI (European Telecommunications Standards Institute) was more about pragmatics than theory. In particular, in the presentation I summarized how the application of model-based testing in standardization differs from applying it in, say, in-house software development.

Currently there are so many small workshops and conferences touching model-based testing (e.g. MBT, A-MOST, MoDeVa, FATES, M-TOOS) that one could guess some consolidation will take place in the future. Maybe a conference dedicated to model-based testing..? Whatever “model-based testing” then means. I proposed in the workshop that one interpretation of “model-based testing” in the research world could be “algorithmic methods for black-box testing that are based on behavioral models of systems under test”, and this was received quite well.

Posted in Conferences and Events | No Comments »

Model Driven or Model-Based, what’s the difference?

March 23rd, 2007 by Antti Huima

I guess it’s fair to say that during the last five years “model-based testing” has become some sort of a recognizable buzz-word in the QA world. But at the same time, so many different notions and ideas have been attached to the phrase that any particular meaning needs to be explained separately. This was actually one of the main problems that the discussion about practical deployment of model-based testing at the M-TOOS workshop (Portland 2006) identified.

For example, some companies mean by “model-based testing” the modeling of testing architectures—this is related often to the UML 2 testing profile (U2TP) that is meant exactly for that. Other people think that it is “model-based testing” when your requirements are described in some sort of a graphical model, and then you derive tests manually and execute them by whatever means afterwards. And the idea that you create “user scenarios” as finite state machines and walk randomly around them is also known as “model-based testing”.

This is why we have always held to the phrase model driven testing. By this phrase we underline our view that model driven testing is about the complete automation of test design that is based on an executable-level system model. This is really test design that is driven by models, maybe in the same way as most cars are driven by people (yet at least): the system model is the elevated, sole source for automatically generated testing logic.

The drawback of this “model driven testing” term is that it is not as good a buzz-word! So we now and then characterize our approach more generally as “model-based testing” just to be able to log into the thought processes of our context groups. It’s basically like calling a sports car a transportation device. Not dishonest or true by any means, but a bit vague. (Of course, these kind of issues surface within any emerging, young concept spaces.)

Posted in Uncategorized | No Comments »

Archives

Categories

Meta

Conformiq Qtronic vs. Conformiq Test Generator

ICST Call for Contributions

Challenge to Our Competitors: Provide Free Evaluation Downloads

Finnish Student Blogs About Qtronic

TESTCOM/FATES, Tallinn, Estonia

Model Driven Testing Tutorial at TTCN-3 User Conference

Toolapalooza 2007

Greetings from Portugal

Model Driven or Model-Based, what’s the difference?