There are managers who hate testing. They think testing is just a cost sink, and that if their software engineers would be clever enough and would follow clean and rigorous processes there wouldn’t be any need for testing. And they might be right. It is not beyond imagination to envision a software process that simply doesn’t let bugs through. It could be a combination of rigorous review and computerized, algorithmic verification approaches. I think it’s a great vision and worth of striving for. Today, the cost of implementing such a process can be much higher than the business benefits it provides, but it’s definitely an exciting goal.

However, there is a fundamental reason why testing is not likely to go away at all, and that is the non-transitivity of proofs of quality. Testing is needed because it is a way to assess the quality, functionality and robustness of a product without having to make a reference to the process by which the product was produced. Company A might have a rigorous software process in place, but how can Company A make a compelling argument to Company B that their OEM component does not require acceptance testing because their processes are so good? Company B cannot take significant business risk that a component they purchase from Company A will malfunction just because Company A claims that they have watertight bug prevention processes. Process excellence is to some extent non-transitive and it cannot be proven to a third party. (And forget those ISO and CMMI certificates. They tell nothing about the concrete details of software quality.)

The same problem can be seen even inside individual companies. Testing tends to be non-transparent, so that managers cannot easily assess if testing teams did actually test what needs to be tested, and if they did that to an extent that is required to curb all the significant business risks. Testing and re-testing is needed because it is very hard to extend proof of adequate testing (and verification and validation) to a third party. This shows also in practice; we have seen many cases where a “fully tested product” has had lots of residual defects that have been exposed when an alternative testing process has been put into place (when it’s us, that is system model driven testing :)). It’s also an economic problem, as normally testing teams or testing companies do not want to give a guarantee or warranty that the product they tested is free of manifesting defects.

Researchers have looked to ways to make quality proofs transitive, though. One such approach is proof-carrying code. It’s not practical today, but one day it might be an essential part of the future’s software ecosystem.


Comments are closed