Tom Gilb is a freelance consultant. Born in California, he has been a resident of Norway since joining IBM, there, in 1958. He has
the Department of Defense as his favorite worthy cause.
Software Inspections are now widely known to the software industry, but most organizations do not make the most of them. This is because many people misunderstand and misinterpret Inspection. This article aims to provide some direction as to how to get the most value from your Inspections.
Given adequate management support, Inspection can quickly be turned from the initial chaos phase (20 or more major defects per page) to relative cleanliness (two or fewer major defects per page at exit) within a year. To understand Inspection, two points must be clear:
Improving Inspections
Following are some key tips about how to improve your Inspection process and how to begin to achieve the kind of results
cited above.
Know Your Purpose for Using Inspection
•
Don't misuse Inspection as a
"clean-up" process. Use it to motivate, teach, measure, control quality, and to improve your
processes. Most people seem to think Inspection is for cleaning up bad work, bugs, and defects. However, the greatest payback comes when
Inspection improves future work, i.e., reduces defect injection. Ensure that your Inspection process fully supports the aspects of
teaching and continuous process
improvement.
For continuous process improvement, integrate DPP into conventional Inspections (see [2] for specific details). DPP
needs to be practiced early. CMM Level 5 is too important to be put off until later—you need to do it from the beginning.
•
Plan Inspections to address your Inspection
purposes. I have listed over 20 distinct purposes for using Inspection,
e.g., document quality, removing defects, job training, motivation, helping a document author, improving
productivity, and reducing maintenance costs [3, 5]. Each Inspection will address several of these purposes to varying degrees. Always be aware which purposes are
valid for your current Inspection, and formally plan to address them,
e.g., by choosing checkers with relevant skills and giving
them appropriate checking roles.
Measure Inspection Benefits
• Measure your benefit from using
Inspections. Inspection should always be highly profitable, e.g.,
10-to-1 ROI. If not, it is time to adjust the Inspection process or stop it. Benefits to be measured include rework costs, predictability, productivity, document
quality, and ROI [1].
Make Intelligent Decisions on What You Choose to Inspect
• Use Inspection on any technical
documentation. Most people think Inspection is about source code inspection.
It started there in the early 1970s; however, once you realize that Inspection is not (in the medium term) a clean-up
process, you can use it to measure and validate any technical documentation—even technical diagrams.
• Inspect upstream first. By the end of the 1970s, IBM and Inspection-method founder Michael Fagan recognized that defects
and the profitable use of Inspection lay upstream in the requirements and design areas. Bellcore found that 44 percent of all bugs
were due to defects in requirements and design reaching the programmers [4]. If your systems start with contracts, management
and marketing plans, you must start your inspection activity there, where the problems start.
One of the most misunderstood dictums to come from the early inspections was
"no managers." Wrong! That was in the
days when Inspection dealt only with source code. Management inspections may be the most useful you will ever do. I have
extremely positive experiences using Inspection with top managers on contracts, marketing, and product development plans.
• Check the significant portions of the material—avoid checking
commentary. Most organizations waste time checking
nonsignificant document areas that do not translate into
a final product; defects in such areas cannot trigger
major consequences. The result of this indiscriminate checking of trivia, at an optimum rate, is 90 percent minor defects and a 90 percent waste of time. It is
like checking comments for 90 percent of the time instead of real code.
We have found that it pays to have a general technical documentation rule that technical authors must distinguish between
text (or diagrams) that can translate to serious downstream costs (noncommentary or "meat") and less-important areas (commentary
or "fat"). This distinction can be done, for
example, by using italics for the fat. Some of our clients have even created
Microsoft Word macros to count the volume of noncommentary text and print it on the first page. Of course, the checker is allowed to
scan and reference the commentary words but is not obligated to check them against all sources,
rules, and checklists. It is not worth it.
• Use sampling to understand the quality level of a document.
It is neither necessary nor desirable to check all pages of long
documents. Representative samples will probably tell you
whether a document is clean enough to exit at, for example, 0.2 majors
per page maximum remaining.
The main purpose of Inspection is
economic—to reduce lead time and people costs caused by downstream defects—not
primarily to clean bugs. As in Harlan Mills' IBM "Cleanroom" method, bugs should be cleaned up or avoided
using disciplines such as Watts Humphrey's Personal Software Process (PSP), Structured Programming,
and Continuous Improvement and Verification. If all this works as it should, cleaning is
unnecessary, and sampling tells you if it is economically
safe to release the document. Perfection is not required—it costs infinite resources and is dangerous as a
guiding concept.
• Inspect early and often while a document is being
written. Inspection after a large (100 pages or more) technical document has
been "finished" is a common, bad idea. If the process that generates the document is faulty, discover it early and put it right. This
saves time and corrects bad processes before they damage your
schedule too much.
Focus on Finding the Majors
• Check at your organization's optimum rates
to find major defects. This is the big one! Most everybody, including so-called
teachers of Inspection, manage to miss this point. Or worse, their suggested checking rates are
10 times optimum speed. Optimum checking rate is not optimum
reading rate. Checking in real Inspections involves checking a page against all related documents. This
can involve up to 10 or 20 source documents of large size,
checklists, and standards. You have to check a
single line against many sources, and it takes time.
Adequate Inspection statistics can prove your employees have a
clear, dramatic, and consistent optimum checking rate
on specific document types. This ranges between 0.2 and 1.8 pages of 300 noncommentary words per checking hour.
At Raytheon, it was about 20 plus or minus 10 lines per hour (0.3 pages). Unfortunately, in spite of their own data, Raytheon suggested
rates of about 100 to 250 lines per hour. This was probably because they had finite deadlines and did not understand sampling.
As the checking speed moves toward an optimum speed for effectiveness of finding major defects, the curve for
optimum checking rate moves dramatically upward in terms of major defects identified per logical page. The optimum may seem slow,
but considering the amount of checking you have to do, it is fast. The main point is that there is a best speed
at which to check, and you will be operating at
low checking productivity if you fail to heed it.
Note that the optimum checking rate applies both to the checking carried out
before and to the optional checking carried out
during the logging meeting. This second check will produce roughly an additional 15 percent defects. You do not need
this extra checking if the document is found clean enough to exit as a result of initial checking sampling or if it is so polluted
that you have to do a major rewrite anyway.
• Define a major defect as "possible larger costs
downstream." It does not matter if a defect is not precisely a bug or if it is visible
to a customer. If it can potentially lead to significant costs if it escapes downstream, classify it as a major and treat it with
due respect.
You can help people identify majors by using checklists that specify how to find
them. (Note: checklists are only allowed to help interpret the "rules," which are the official standards for writing a given document, and which define defects). Use
symbols: "M" for major and "m" for minor after the
checklist questions or rule statement. I also often find it useful to use "S"
for super major or showstopper (a defect where the downstream effect could be an order of magnitude bigger than an average
major). Super majors can be highlighted for management attention.
• Log only major defects. This helps avoid the "90 percent minor" syndrome that often hampers Inspection. Employees
waste time identifying 90 percent minor defects unless strongly
redirected. There are 18 tactics that shift your focus from minor
to major defects ([2], pp. 75-76). For example, allow only ideas for finding
majors onto the rules and checklists, log only majors
at a meeting, and calculate ROI for Inspections only
based on majors. A clear message must be given to not waste time on
minor defects.
Apply Good Practice When Leading Inspections
• Use serious entry conditions, e.g., numeric quality of
sources. We are in such a big hurry to waste our own
time. Many do not have the discipline to set up and respect entry conditions that prevent wasting time—but you
must.
One of the most important entry conditions is to mandate the use of upstream source documents to inspect a
"product" document. It is a mistake to try to use the experts' memory abilities instead of updated,
Inspection-exited source documents. It is a farce to use source documents with the usual uncontrolled, uninspected, unexited, 20 or more major defects per page
to check a product document. It is silly to allow a product document author to use a bad quality source document to generate
a product. Inspection does not have to repeat that
silliness.
You can ascertain the state of a source document through inexpensive sampling. A half day on a few pages is a small price
to pay to know the state of a document that could destroy the quality of all your
work and your project.
Another serious entry condition is to do a cursory check on the product document and
return it to the author when it is anything less than a quality piece of work bursting at the seams to exit (based on
a cursory check that reveals few remaining defects). For example, if while planning the Inspection, the team leader performs a
15-minute cursory check that shows up a few major defects on a single page, it is time for a word with the author in private. Pretend the document was never
seriously submitted. Certainly do not waste the time of your team to confirm shoddy work.
In short, learn which entry conditions you need to set, then take them seriously. I would like to see management lead
by understanding the importance of this instead of ignorantly thwarting engineers' attempts to do reasonable work by
insisting Inspections proceed when the entry conditions have not been met.
• Make sure you have excellent standards to identify defective
practices. Inspection requires that good work standards
be in place. Standards provide the rules for the author when writing technical documents and then for the Inspection process to
subsequently check against. Standards must be built by hard experience; they need to be brief and to the point, be monitored
for usefulness, and must be respected by the troops. They must not be built by outside consultants or dictated by
management. They must be seen as the tool to enforce the necessary lessons of professional practice on the unwary or unwilling.
• Check against source and kin documents; check them for
defects, too. Because of potentially poor quality control practices
and craftsmanship, and because Inspection is imperfect on first pass (30 percent to 80 percent effective), focus on major
defects that persist in source documents used to produce the document under
Inspection, and in kin documents derived from the
same source documents. For example, if a functional specification was the product document requiring Inspection, there should
be a requirements document as one of the source documents and a testing document as one of the kin
documents.
Most people overfocus on the product document, i.e., the document you are inspecting and evaluating for exit. You
should probably be finding 25 percent of your total defects
external to the product document.
• Use the optimum number of people on a team to serve the current purpose of Inspection,
e.g., effectiveness, efficiency, and training. For 13 years, one large U.S. telecommunications company
had 12 to 15 people on each inspection team because each "had"
to be there to protect territorial interests. There seemed to be no motivation to cut these costs.
The number of people who are needed on a specific Inspection team is a function of your purposes. Monitor the results
of varying team sizes to discover your optimum. If you measure your own Inspection experiences, you will find that
effectiveness at finding major defects uses four to six people, efficiency (effect over cost) needs two to four people, and only teaching as
a purpose justifies larger numbers.
• Allocate special defect searching roles to people on the
team. Each person on an Inspection team should be finding
different defects. Much like a coach on a ball team, the Inspection team leader should assign specialist roles to team members,
e.g., identify time and money risks, check against corporate standards for engineering documentation,
and check security loopholes.
• Use checking data (such as pages checked, majors found, time used, and checking rate) from individual checkers to decide whether it
is worth holding a logging meeting. Older types of inspection plunge into the logging meeting without forethought, and
consequently waste a lot of time. We have developed a process of logging meeting entry evaluation before going ahead with the logging
meeting. We collect data from checkers about checking rates and major-issue density.
(To avoid personal conflict, issues—not
defects—are logged during the logging meeting. An issue may or may not become a
defect.) Based on this data, we make a series of
decisions about the logging meeting. Most critical is whether a meeting is
necessary. Other decisions include whether to log minors,
whether to continue checking, and what is the likely optimum checking rate.
• Use the individual checkers' personal notes instead of proper meeting defect logs when the major issue volume is (nonexit level) high,
or when there is a large number of minor defects.
Checkers should not be required to use any particular method to make notes
during checking. But most of them mark a paper document (some use an electronic document) with an underline or circle, or they
highlight offending words. It is important that they
also note, against offending words, exactly which rule was broken (the issue).
To note major or minor is less important if all issues found are majors.
Whenever there is a higher volume of issues than would indicate
allowable exit, you can happily, with author agreement,
return these "scratchings" to the author rather than pedantically log them like good bureaucrats.
Authors need to rewrite and resubmit in these cases, using this information to correct their work
processes (usually to take sources and rules more seriously).
• At logging meetings, avoid discussions and avoid suggesting
fixes. Inspection is not for talkers and quibblers—it is for
professionals committed to making maximum, meaningful progress on the project.
You can have a good time, but not by idle gossip and
insults. You are there to measure, not to wear each other out, or get drowned in unprofitable bureaucratic games.
• Use serious exit conditions, e.g., "maximum probable remaining major defects per page is 0.2 for
exit." Exit conditions, if correctly formulated and taken seriously, can be crucial. It is ridiculous and sad to have the customary "vote" to accept a document
when the logged defects are fixed—this completely ignores the known factor of
remaining unfound defects, which is computable and
verifiable from past data and experience.
Remember that Inspection processes, as other testing processes, have a maximum effectiveness for a single pass in the range
of 30 percent to 88 percent of existing defects. If the maximum probable remaining defect density is a high-quality low count,
e.g., 0.2 majors per page, it matters little if the detected defects are removed; the document is clean enough (economically speaking)
to exit without fixing those identified.
If defect density is high, e.g., 20 or more majors per page (quite common), undetected defects, at say 50 percent
effectiveness, are more than enough to make exit uneconomical.
Ten majors remaining per page in a 100-page
document would result in 9 x 10 x 100 hours of additional project work to clean them up by testing and discovery in the field. It costs an
order of magnitude less to find them now.
Admittedly, it is only the lesser of two evils—you should wish to
prevent them using DPP rather than have
to clean them up early using DDP.
Management must understand the large-scale economics of this, making clear policy about the levels of major defects per
page that will be allowed to escape ([2] pp. 430-431). The consequences of poor policy here should be deducted from
management's pay!
Ensure You Have Provided Adequate Training and
Follow-up
• Give Inspection team leaders proper training, coaching after initial training, formal certification, statistical follow-up, and if
necessary, remove their "license to
inspect." Proper training of team leaders takes about a week (half lectures and half practice). Formal
Inspection team leader certification (an entry condition to an Inspection) should be similar to that for pilots,
drivers, and doctors—based on demonstrated competence after training.
Leaders who will not professionally carry out the job, even if it is because their supervisor is pressuring them to cut
corners, need to have their license revoked. If you take professional leadership seriously, your players will take Inspection seriously.
Ensure there is an adequate number of trained people to support your Inspections—at least 20 percent of all professionals. Some
clients train all of their engineers.
Give Visibility to Your Inspection Statistics
and Support Documentation
• Put your Inspection artifacts on a company Web
site. If you have an Intranet, all relevant Inspection artifacts, standards,
experiences, statistics, and problems should be placed on a corporate-wide site as soon as possible.
• Build or buy an automated software tool to process Inspection basic
data. Use automated software tools to capture summary data
and to present trends and reports [7]. Inspection quickly generates a lot of data that is fundamental and useful to managing the
process. It is vital that good computer support be given early so the process owners and management take the data seriously and so
that early champions are not overwhelmed.
The key distinction between Inspections and other review processes is the use of data to manage them. For example,
optimum checking rates must be established early and updated as they change through continuous improvement. It also is vital to
statistically see the consequences of inadequate exit levels (too many major defects floating downstream), which then are caught with
expensive testing processes. Locally made spreadsheet software is a good
start-up tool.
• Plan Inspections well using a master
plan. We have developed a one-page master plan ([2] p. 401) that goes far beyond the
conventional "invitation." We document the many supporting documents needed, assign checkers their special defect-searching roles,
and carefully manage rates of checking and the total checking time needed. We establish the formal purpose(s) of this specific
Inspection, which vary. We establish a team numeric stretch goal for this Inspection and a strategy to help attain it. A good master
plan avoids senseless bureaucracy and lays the groundwork for intelligent Inspections.
Continuously Improve Your Inspection Process
• Use Defect Prevention Process on the Inspection
process. Finally, recognize that systematic continuous improvement of the
Inspection process is necessary. Initially, this is required not only to improve the process but
also to learn the Inspection process properly and to tailor it to your
organization.
Acknowledgment
I thank Lindsey Brodie for her editing and assistance on this article.
About the Author
Tom Gilb is a freelance consultant. Born in California, he has been a resident of Norway since joining IBM, there, in 1958. He has
the Department of Defense as his favorite worthy cause.
E-mail: Gilb@ACM.org
References