Software Quality Journal 3, 59-44

News

Section

Book Reviews Editor, UK Margaret Ross, Southampton Institute, Southampton, Hampshire SO9 4WN, UK Book Reviews Editor, US Taz Daughtry, Babcock and Wilcox, PO Box 785, Lynchburg, VA 24505-0785, USA

The news section of Software Quality Journal will contain reviews of relevant books, articles and conference proceedings. Although suggestions of relevant publications for review will be welcomed, unsolicited reviews will not be. As the section evolves, it is hoped that it will develop to accurately meet the readers’ needs.

Book Reviews Testing Computer Software

Cem Kaner, Jack Falk and Hung Quoc Nguyen, Van Nostrand 2nd edition, 480 pp

Reinhold,

New York (1993)

This book is based on the authors’ experience in testing and managing testing, for a publisher of personal computer software. It was originally written as a training guide for their staff. The structure and layout of the book reflects this training approach. The first five chapters discuss the fundamentals of testing. The next section (seven chapters) covers specific testing skills; these often discuss issues introduced in the first section in more depth. The final section (3 chapters) covers management issues: managing the testing process and managing test teams are both discussed. There is an appendix of one hundred pages on common software errors. The other major difference from standard texts, such as those of Myers, Beizer and Hetzel, is in the emphasis on testing software developed for sale, typically to users of personal computers. The authors point out that for such products, it is not possible to agree a specification with potential purchasers and that any design specification may be subject to changes at any time if a competitor brings out a product with features that appeal to purchasers. The development team, including the testers, must be able to cope with such changes and turn out software of satisfactory quality. In this situation, the authors argue for an emphasis on functional (or black box) testing, rather than on structural (white box) testing. It is assumed that the programmers will have carried sufficient testing during development, using both white and black box techniques, so that the product is ‘stable’ enough for the testing team to test it effectively. The first stage of testing is to check that the product is sufficiently stable. The Black box procedures differ from those used in mainframe software development, for example, in the use of Beta testing of products. For mainframe based projects, particularly those which are safety-critical or which relate to 0963-9314 0 1994 Chapman & Hall

60

News section

major financial software, the authors recommend the use of the standard text books. However, they point out that much of the material in this book is more appropriate than the standard texts for those carrying out acceptance tests on such projects. A brief outline of the content of some of the chapters may help readers to decide whether this book would be useful in their environment, to supplement or replace the standard texts. This outline is concerned mainly with material that does not appear in the standard texts. Other chapters discuss issues such as test case design, text planning, test documentation and testing tools. The discussion is generally similar to that of the standard texts, but is developed in the context of testing software for personal computers. Chapter 5 presents a typical Problem Report form, which may be paper based or on-line. Problem reports may come from technical support staff, beta testers and customers, as well as testers and the difficulties of dealing with reports from these sources is discussed. There is a good, detailed, discussion on how testers can complete these forms effectively, so that the problem has a high probability of being corrected. Chapter 6 discusses the objectives, and a possible implementation, of a Problem Tracking system using these Problem reports as input. The possible users of the tracking system are considered, and the ways in which each type of user could use the data (and attempt to misuse it) are discussed. Chapter 8 considers the testing necessary to check that the software will run on a wide range of personal computers, with a wide range of peripheral devices. A detailed consideration of testing the software with a wide range of different printer is used to consider the issues involved in some depth. Chapter 9 considers the testing necessary after a product has been localized, i.e. adapted for use in another country. This may involve translating text to another language as well as adapting to different hardware and operating systems. Adapting to a different culture may also be important. Some cultural issues are easy to identify and to adapt to, for example date formats and sorting orders. Others, such as the use of a decimal comma rather than a decimal point are a frequent source of problems. Other issues, such as the use of culture-bound graphics (e.g. icons used in a GUI) may not be apparent to the adaptor or the tester. The user documentation is an important part of the product, and includes reference manuals, installation guides and on line help facilities. Chapter 10 discusses the testing of this material. One obvious purpose for this is to correct errors in this documentation. However, the authors state that this testing often reveals serious errors in the programs. Chapter 14 discuss the legal consequences of Defective Software. Many interesting issues are raised, but the discussion is based on American Law, mostly relating to general product liability, not specifically to software products. The final chapter discusses the management of the testing group. The discussion includes the role of the testing group and its relationship to Quality Assurance teams, project management issues related to testing and personnel issues (such as the attributes and skills of a good tester). Software Quality management issues are raised in a number of chapters, mainly those relating to managerial issues. A brief introduction to Total Quality Management is given, with references to standard textbooks. However, the authors’ claim that ‘if you run a typical software text group in a typical American Software company, you won’t have TQM, but you can adopt a workable, narrower vision and provide tremendous value to the company’. This book gives the authors’ version of such a ‘workable, narrower vision’. Quality management standards, such as IS0 9001, are not mentioned, but some information on

61

News section

ANSI/IEEE standards relevant to testing is given in the chapter on legal issues. However, the author of this chapter claims ‘The ANSI/IEEE standards should apply to consumer software testing, but very few consumer software test groups (I know of none) find these standards relevant to their work. Few consumer software testers have read these standards, and of those testers, very few (I know of none) consider the standards useful. To my eyes, these documents assume or call for an excessive level of administrative overhead without providing useful guidance for prioritizing tasks and eliminating less essential text cases. I assume these standards are more applicable to larger projects’. The authors’ views on quality are summarized in the following quote: ‘The quality of a great product lies in the hand of the individuals designing, programming, testing and documenting it, each of who counts. Standards, specifications, committees, and change controls will not assure quality, nor do software houses rely on them to play that role. It is the commitment of the individuals to excellence, their mastery of the tools of their crafts, and their ability to work together that makes the product, not the rules’. I have heard people from software hosues in the UK expressing similar views. In summary, this book meets its aim of providing a training guide for testers of consumer software very well. In more standard software development environments, it will not replace the standard texts, but it could be a useful supplementary text. References Beizer, B (1990) Software Testing Techniques,Van Nostrand Reinhold, New York. Hetzel, B (1988) The Complete Guide to Software Testing, Wellesley: QED Information Servies. Myers, G. J (1979) The Art of Software Testing, John Wiley, New York. GEOFF STAPLES

Southampton Institute Software Process Technology Edited by Jean-Calude Demiame, Springer-Verlag

(1992) ISBN 3-540-55928-0, 253 pp.

This book, volume 635 of the Lecture Notes in Computer Science series, consists of 33 contributions to the Second European Software Process Workshop (EWSPT ‘92), held in Trondheim, Norway in September 1992. Many of the papers are only of 2-3 pages and, as is typical in such a compilation, they are of widely varying tone, detail, and focus. Nearly threequarters are from academics. Process modelling is described from the point of view of such approaches as structured analysis, social interactions, object orientation, and within specific modelling formalisms and development environments that will have limited applicability to non-users. One such formalism was applied to reveal inconsistencies and ambiguities in the IS0 9000-3 software quality management guidelines. Software Metrics: A Rigorous Approach Norman E. Fenton, Chapman & Hall (1991), ISBN O-412-40440-0, 337~~. With five other contributors, Fenton draws upon work done for the ESPRIT-funded METKIT (Metrics Education ToolKIT) project and makes a strong case that measurement-done more rigorously and more imaginatively-must play a more significant role in software engineering.

Following a solid chapter on measurement theory, he considers the attributes of software development products, processes, and resources that can lead to successful prediction systems. We next encounter, in turn, experimental design, measurement validation, and data collection. There is practical advice on setting up a measurement program, on improving upon existing estimating models, and on understanding a large numer of specific current measures being applied to both internal and external product attributes. Throughout there are excellent chapter-ending summaries, scattered exercises, and good use of graphics for illustration and emphasis. References are copious and to the point. One treasure is the concluding 352-item annotated bibliography, done with the same verve as the body of the book and having no obvious oversights. Final wisdom: ‘Put objectives first. ’ ‘Measurement must be supported by theory.’ ‘Have realistic expectations.’ Software Quality: Theory and Management Alan C. Gillies, Chapman & Hall (1992), ISBN o-412-45130-1, 250 pp. The author states that the intends this work to bridge the gap between software engineering and quality management. Though used at his institution as reading in the Masters program, it is not a textbook. Most of the relevant topics-quality attributes, metrics, development tools, quality standards-are treated, but often at a gallop. One can scarcely turn a page without encountering a table, chart, or sketch (although some of the latter are a bit too cute). These aids certainly help make the material more palatable, but they also reduce the total contents to the volume to slightly over that of an extended monograph. Many quality topics are touched upon, often without a strong tie to software specifics: sayings of the gurus, SPC techniques, quality circles. Intriguing software-related topics range from ‘Locally defined quality modelling’ to ‘Is TQM appropriate for software development? to ‘Corporate IT effectiveness.’ Unfortunately, Gillies chooses to devote over a dozen pages to IS0 9001 in general, but less than a page of text and one table to the software-specific IS0 9000-3. Two extended case studies explore the need for a quality culture in both the hardware and software worlds. Software Fault Tolerance: Achievement and Assessment Edited by Manfred Kersken and Francesca Saglietti, Springer-Verlag ISBN 3-540-55212-X, 243 pp.

(1992),

Eight contributors from Germany, Italy, and the UK offer 14 chapters on fault-tolerance research done within ESPRIT’s REQUEST project for the assessment of quality and reliability in software processes and products. Practitioners not specializing in the topics will find it hard sledding through detailed discussions of failure dependencies, software diversity, and recovery block programming. Read the opening overview and concluding chapters to see if you need to probe any further. Independent Verification and Validation: A Life Cycle Engineering Process for Qualtiy Software Robert 0. Lewis, Wiley-Interscience (1992), ISBN o-471-5701 1-7, 356 pp. The author, Program Manager of Air Defense Systems for General Research Corporation,

fails to

News section

63

rise above the minutiae of US Defense Department-specific projects and thus does not deliver on the stated aim of serving as either a reference handbook or textbook. For instance, the 26 references include no fewer than 15 military standards or regulations, and all but three of the remainder are over ten years old. Tables with a ‘Required by Congress’ column are also unlikely to be generalized to other settings. There are some interesting discussions on effective planning, criticality assessment, and tailoring high-cost programs to medium- and low-cost options. The book concludes with eight IV&V case studies, each presented in only 2-3 pages and of correspondingly limited usefulness. Software Systems Engineering

Andrew P. Sage and James D. Palmer, Wiley-Interscience

(1990), ISBN 0-471-61758-X,

520 pp.

A methodical tour of the landscape by two George Mason University faculty members, this book has predictable sections with titles such as ‘Models for the software development lifecycle’ and ‘Systems requirements identification and software requirements specification.’ Yet, along with material on design, coding, testing, and system integration, there are also extensive treatments under the headings l ‘Software reliability, maintainability, and quality assurance’ . ‘Prototyping, reusability, and reverse engineering’ l ‘Management of software systems engineering process’ and l ‘Software cost and value models.’ Written for a first course in software engineering, the book ends each of its ten sections with several pages of ‘Problems’, which are typically suggested topics for short essays or undergraduate research papers. A concluding 15-page reference list is not annotated but is keyed to a page-long topical outline.

Abstracts Process improvement and the corporate balance sheet Raymond Dion (1993) IEEE Software, 10 (4), 28-35.

Dion updates the story on Raytheon’s quality improvement initiative: an effort begun in mid-1988 that showed by the end of 1992 a cumulative $15.8 million saving, primarily from a four-fold reduction in the costs of rework. Other ‘bottom line’ conclusions: a $7.70 return on each dollar invested in process improvement; a doubling of productivity in terms of equivalent delivered source instructions per man month; and impressive schedule and budget improvements. (One project earned almost $10 million - not included in the savings calculation - as a schedule-incentive bonus.) What worked? For one thing, the 400 software engineers involved received a total of 564 courses in 1992, ranging from 8 to 64 hours each of on-the-job time. Quality control in software documentation:

Franz Lehner (1993), Information

measurement of text comprehensibility

& Management 25 , 133-46.

64

News section

The author describes and evaluates eight methods of measuring documentation quality and seven tools for computing some of these measures. His own Readability Measuring System prototype (running on a Macintosh) is applied to four commercially supplied user manuals - the German and English language versions of a Microsoft Word manual - as well as two other German manuals for bookkeeping packages. Although the latter were recommended as ‘easily readable’, Lehner found all four ‘poor’. The tool uses the Cloze procedure of blanking out words and seeing if the reader can correctly fill in the blanks. The German language texts, when surveyed by undergraduate business infomatics majors, yielded 50-53% correct substitution of blanked words, while the English text averaged 37%. An investigation of the Therac-25 accidents

Nancy G. Leveson and Clark S. Turner (1993), Computer, 26(7), 18-41. An admirable piece of investigative and narrative work, this article moves the dramatic failure of software-controlled radiation therapy devices from the realm of the anecdotal to a full-blown case study. It represents a sobering story of unexamined incremental product evolution, a highly personalized programming effort and the massive safety analysis (unfortunately done retrospectively) necessary to understand the risks involved. There were actually multiple software flaws, often appearing only under circumstances of hardware configuration and operator data-entry timing that proved maddeningly difficult to reproduce. The authors offer lessons learned for both system and software engineering, as well as regulatory oversight.

Conference Third International

Detai Is Conference on Software Quality

sponsored by the Software Division of the American Society for Quality Control, Nevada, October 1993.

Lake Tahoe,

Two thick volumes - one of conference proceedings and the other of tutorial notes - cannot begin to capture the insights shared by some 300 participants at this annual event. Again this year, approximately one-quarter of both presenters and attendees came from outside the USA. One track was organised by the Union of Japanese Scientists and Engineers. Presentations ran the gamut from introductory to advanced research; their topics ranged from process assessment and improvement to measurement, product and personnel evaluations. The ASQC will hold its October 1994 conference in the Washington, D.C. area in cooperation with the local chapter of the Software Process Improvement Network. For more information, contact Sue McGrath, SAS Institute, Inc., SAS Campus Drive, Cary, North Carolina, USA (phone: l-919-677-8000 ext. 7032, fax: 1-919-677-8224, Internet: [email protected]). She can also provide information on the availability of Proceedings from previous conferences.