Software Test as End Item Inspection

Manufacturers of tangible goods have been moving away from end-item-inspection for quite some time. This is especially true with more complicated items since end-item-inspection can rarely exercise all aspects of even small consumer goods such as appliances let alone something as large and complicated as an automobile. Ever since the work of Edwards Deming it has been recognized that end-item-inspection methods do not result in a quality product. Given this broad recognition that end-item-inspection is futile for ensuring or improving product quality, why is it still relied on by many software development organizations?

There have been a number of initiatives to improve software quality by working to improve the software development process. The software capability maturity model (SCMM) is one of the better known of such general efforts with predictive methodologies such as the Rational Unified Process (RUP) being a methodological instantiation of the SCMM recognized best practices. Agile methodologies take a different approach of iteratively building the project in small steps that must work but the idea is again that the project's quality is continuously checked. In both cases, project quality isn't left as an afterthought; relying on some test group to run a few tests that supposed verify that the project actually does what it is supposed to do.

At best end-item-inspection methods can only show that a product does what it is supposed to do. Destructive end-item-inspection of tangible goods can show that a particular product will not fail until the design limit under test is exceeded. For software the comparable activity is testing to ensure that the program doesn't break under expected usage and even expected misuse. The problem here is the word "expected." The developers will (hopefully) consider misuse of the program including attempts to hack the program and take steps to ensure that the program continues to function correctly in spite of such misuse. Unfortunately at the individual developer level little can be done except to anticipate misuse and include code that mitigates expected misuse. Hackers have a nasty habit of finding unanticipated attack vectors that still allow them to crack the system. End-item-inspection is useless at finding such flaws or other more subtle problems that only occur infrequently since they are due to a low probability interaction between system components.

This is the crux of the problem with attempts to "test quality into the code." Testing only verifies that when specific preconditions are met, the program under test behaves as expected. End-item-inspection is totally unsuitable for finding design flaws (the infamous, "Works as designed.") and unexpected attack vectors. End-item-inspection can at most verify that the functionality that is supposed to be present is actually "there" and that expected misuses are handled. Such an approach will catch the worst of the flaws and system instabilities but the low probability interactions will rarely be created and, even if they are, there is little likelihood that the problem can be recreated so that the root cause can be determined and fixed unless the test was specifically designed to look for the flaw.

Back in the early days of software development, end-item-inspection was not recognized as being ineffective. While manufacturers of tangible goods looked to process improvements to improve product quality software went through a different maturation process. Programming languages and development environments evolved from assembly language on punch cards or paper tape to high level languages like Java developed in integrated development environments with context sensitive editors and modern configuration management techniques. This evolution in how software was written did a reasonable job of eliminating the low-level errors that plagued earlier developers but the expectation was still that end-item-inspection of the final program would be sufficient.

As mentioned previously, initiatives such as the software capability maturity model attempted to look at improving the process of developing software. Unfortunately these efforts were typically tied to the high ceremony, rigid, predictive development processes of the defense and aerospace industries. My experience has been that any attempt to introduce such an approach to a commercial development effort is quickly dismissed with the bland statement of, "We don't have time for that nonsense." Also, as mentioned previously, agile methodologies have attempted to make continual process integration and demonstration of working code to be part of the development process. This works within the problem domains that are suitable for agile methodologies (see my earlier post on why agile methods don't scale). Larger and more complicated efforts or even small complicated efforts for which no suitable design pattern exists need the engineering discipline of a predictive methodology even if they don't need the high ceremony associated with predictive methodologies.

Given a large or sufficiently complex software project, software test using end-item-inspection is an exercise in futility. At best the testing effort will show that the project behaves as expected within a narrow range of use. The real world will rapidly begin finding the flaws that result from unexpected use. This behavior was true with tangible goods that relied on end-item-inspection and it will remain true of software projects until users and developers demand something better.

Cheers,
Dave