Distributed Test Generation

As detailed in a blog post I wrote a couple of months ago, the core of Conformiq DesignerTM is a custom crafted  semantics driven, symbolic state space exploration  algorithm for test generation from system models (because this is really the only known solution that robustly generates both test inputs and outputs from a system model without user intervention). On a very high level, this algorithm selects which execution paths in the model to expand and to what extent and Conformiq has been carefully tuned our implementation during the last decade to make the operation more efficient, robust, comprehensive, and user tunable.

Because large real world test generation problems are computationally very complex and can take long to process on a single PC, this test generation algorithm has been carefully crafted for fully distributed parallel test generation, allowing users automatically to split test generation tasks across multiple cores and PCs in, for example, a public or private cloud / cluster environment or just a large server. This technology enables users to create computational clusters for rapid test generation that can be shared across the users. However even without access to a cloud or a cluster environment, users can still gain the benefits of parallel test generation with Conformiq Designer because even on a single PC setup it automatically maximizes the use of computation resources available locally on the workstation to speed up the test generation process by distributing computation work over all the processor cores available on the local host. Continue reading

Post to Twitter Post to Facebook Send Gmail Post to LinkedIn

Posted in Uncategorized | Tagged , , , , , | Leave a comment

What is Important When Selecting an MBT Tool?

Interest towards model-based testing has increased quite significantly over the years as people have started to reach limits of traditional approaches and at the same time started to see and understand the benefits that applying MBT can have to the quality assurance function. In this blog post, I’m outlining what is really important when you are selecting and evaluating an MBT tool. Continue reading

Post to Twitter Post to Facebook Send Gmail Post to LinkedIn

Posted in Uncategorized | Tagged , , | Leave a comment

Equivalence Class Partitioning and Boundary Value Analysis as Black Box Test Design Heuristics

Boundary value analysis is a refinement of the equivalence class partitioning method which again is one of the most generally applicable methods for black-box test design. The idea of equivalence class partitioning is to divide the all possible inputs to the system into “equivalence classes”, i.e. sets of inputs that should produce “analogous” results and “work the same”. As a simple example, let’s take a ticketing system where children under age 6 are allowed to travel for free, people under 18 as well as senior people older than 64 pay $10 while adults need to pay $20. For testing this system, we can use equivalence class partitioning to design our tests.

equivalence classes

When the equivalence classes involve numbers, there are decision boundaries between them. These are places where the behavior of the system changes. What makes boundary value analysis an interesting method is that it is widely recognized that values on the boundaries cause more errors in system. Therefore we should be always checking the boundaries as if the system fails, it is likely to fail on these decision boundaries.

In the example above, there are decision boundaries around age 0, 6, 18 and finally around age 64. First, the age cannot obviously be less than 0; children between 0 to 5 years can travel for free, while at age 6 they are charged for $10. When you turn 18 you need to pay the full price of $20. Finally when you turn 64 you get senior discount.  If boundary value analysis is applied to this example, we as an example could get the following tests to test the boundary around 0 and 6:

Test Expected Result Equivalence Class
-2 Reject the input Negative value is invalid
-1 Reject the input The greatest value that is still invalid
0 Travel for free Children under 6 (the smallest value in this class)
1 Travel for free Children under 6
4 Travel for free Children under 6
5 Travel for free Children under 6 (the greatest value in this class)
6 Pay $10 Children between 6 and 17 (the smallest value in this class)
7 Pay $10 Children between 6 and 17

One of the problems with equivalence class partitioning and boundary value analysis is that they are typically explained with simple examples like the one above, while in the real world these methods need to be applied in relation to every equivalence class of the reachable system states of the application that we are testing. For real world systems, carrying this out manually easily become prohibitively difficult.

Conformiq Designer implements automatic equivalence class partitioning and boundary value analysis across the whole system specification (as encoded in a model). The tool is able to generate test suites that cover the equivalence classes and boundary value cases (and if not produce a report of what we failed to cover during the test generation) that at the same time are of reasonable size with full coverage and traceability information. You are relieved from doing equivalence class partitioning and boundary value analysis by hand, a time-consuming and error-prone process that is also very complicated for larger systems.

Post to Twitter Post to Facebook Send Gmail Post to LinkedIn

Posted in Uncategorized | Tagged , , , , , | Leave a comment

Performance of Test Generation

Test generation from system models is computationally very hard: Just generating input sequences that cover all the statements of a system model is theoretically an undecidable problem, meaning it can be never solved completely. This does not mean there couldn’t be an algorithm that handles most of the industrially relevant problem instances, but it gives a hint that it’s a challenging engineering problem producing such an algorithm.

Over the years, one of the single biggest complaints against using computer to generate tests from a system model is that it simply takes too much time or even worse, nothing ever comes out. We’ve had our fair share of these complaints, however, rest assured that scalability issues are taken very seriously and we are constantly investing time into research and development of more efficient algorithmic approaches to automated test design.

In this post I’m detailing a bit of what is happening “under the hood” in the actual test generation core and how Conformiq has evolved our flagship product Conformiq Designer to meet the needs of the industry. Continue reading

Post to Twitter Post to Facebook Send Gmail Post to LinkedIn

Posted in Uncategorized | Tagged , | Leave a comment

ETSI UCAAT 2013

In late October, ETSI organized for the first time a user conference for advanced automated testing – UCAAT for short. This conference is, on contrary to many other academic conferences and workshops on automated testing, dedicated to the practical engineering and application aspects of automated testing including model-based testing, test methodologies, test management and use of test languages. It is a practical and industrial conference that aims to give users from different application domains a chance to meet and share their practical experiences, lessons learned and get advice on the use and application of latest techniques, methods, frameworks and tools. The idea is that users present independent of tool vendors. Continue reading

Post to Twitter Post to Facebook Send Gmail Post to LinkedIn

Posted in Conferences | Tagged , , | Leave a comment