Accred Qual Assur (2006) 11: 647–652 DOI 10.1007/s00769-006-0164-2

David Brynn Hibbert

Received: 6 April 2006 Accepted: 20 April 2006 Published online: 26 July 2006 C Springer-Verlag 2006
D. B. Hibbert (
) School of Chemistry, University of New South Wales, Sydney, NSW 2052, Australia e-mail: [email protected] Tel.: +61-2-9385-4713 Fax: +61-2-9385-6141

MEETING REPORT

The Hitchhiker’s guide to quality assurance

Abstract A report is given on a 2-day symposium “The Hitchhikers Guide to Quality Assurance” held in Brisbane Australia, in 2005. More than 100 representatives of field laboratories attended this educational exercise organised by the Royal Australian Chemical Institute.

Introduction The late British author Douglas Adams (1952–2001) became famous in 1978 when he wrote a radio drama entitled, “The Hitchhikers Guide to the Galaxy.” Books, more radio series, a TV series and recently a Hollywood film followed. The original inspiration was a series of travel guides “The Hitchhikers Guide to. . .,” and the story recounts the adventures of an Earthman, Arthur Dent, who travels through the Galaxy when planet Earth is destroyed to make way for a hyper-space bypass. The search also involves the answer to life, the universe and everything ( = 42) and the ultimate question for which 42 is the answer. This unlikely background suggested a theme to the Royal Australian Chemical Institute (RACI) around which to organize a symposium on Quality Assurance in the Chemical Laboratory. This ran very successfully in 2003 and 2004 in Sydney, and was transported to Brisbane in 2005. Metrology in Australia is served by its National Measurement Institute (NMI); before the NMI’s creation in 2004, chemistry came under the National Analytical Reference Laboratory (NARL). The accreditation organization, the National Association of Testing Authorities (NATA) was the first such body in the World. Although Australia has taken its place in the global metrology in chemistry community, like many countries, there is a great need to introduce basic concepts of metrology to field laboratories: metrological traceability, measurement uncertainty, quality control tools, proper calibration, the role of accreditation, and so on. The lectures and workshops in this seminar were therefore directed at practitioners who might not be sophis-

Keywords Quality assurance . Traceability . Measurement uncertainty . Quality control . Education

ticated metrologists, but all shared a desire to make good chemical measurements. This report gives brief details of the seminar (see Table 1). Further information and copies of the book of the seminar [1] may be obtained from the organisers Brian Kreiger ([email protected]). Day 1 Proceedings were opened by Brian Krieger, the local organizer, who stressed the practical nature of the seminar. It was hoped to bring useful insights and recipes for field laboratories faced with accreditation and quality assurance demands from clients, regulatory authorities and their own desires to provide reliable chemical measurements. Professor Brynn Hibbert gave the opening lecture on metrological traceability. This will be reproduced in a future issue of the journal [2]. The discussion showed that the widespread (and incorrect) belief that quality-control checks with CRMs, of themselves, establish metrological traceability, is an important misunderstanding that needs to be clarified. The availability of CRMs is also an issue in Australia. Being geographically remote, Australia has some of the strictest quarantine laws designed to keep exotic pests and diseases out of the country. In the past, some CRMs have been opened at customs, delayed and generally refused entry. Negotiations between the NMI and the customs department have ameliorated the situation but the question still remains of what to do if a suitable CRM does not exist. (See an editorial in this journal for comments on this problem [3]).

Vanessa Hodgson – Australian Federal Police, Canberra Pieter Scheelings – Queensland Health Scientific Services (QHSS), Brisbane

Chain of Custody: “I swear it was 42 when it left the building; what do you mean its 24 when the lab analysed it?” Establishing continuity of possession, and proof of integrity

Competent People: their impact and input in quality programs and measurement reliability

Ivan Waples - Waples Consulting

Auditing: “Visits from the Vogons, DON’T PANIC.”

Stephen Davies – NMI Adam Cawley – ADSTL Kim Colyvas – University of Newcastle Maree Stuart – MAS Management Consultancy Services

Certification of Reference Materials

Calculating Uncertainty

Statistics for QC

Proficiency Testing

TQM: “Throw away those Peril-Sensitive Sunglasses and relish your business improvement opportunities”

Presenter Eveline Baker – JEB Creative Solutions Pty Ltd

John Eames – LQMS Maree Stuart – MAS Management Consultancy Services

Quality Control: “I found it in the Encyclopedia Galactica: Everything you need to know, but never asked.”

Day 2 Workshop Method Validation

Kirsty Outhred – NATA

ISO/IEC 17025 – 2005: “You don’t need a Babel fish to understand 17025 – 2005” What’s new in 17025 – 2005

How to survive an accreditation visit

Stephen Davies – NMI

Reference Materials: “Zaphod Beeblebrox had two heads, I have one, are we both people?” Choosing the right reference materials

Strategies for effective calibration

Calibration: “The Infinite Improbability Drive: If it passes through every point in the universe how do I calibrate its speedometer?”.

Brynn Hibbert – UNSW

Kirsty Othered – National Association of Testing Authorities (NATA) Pieter Scheelings – QHSS

Panel Discussion

On NATA requirements/guidelines, method development CODEX/AOAC requirements

Adam Cawley – Australian Drugs in Sport Testing Laboratory, NMI, Sydney

Measurement uncertainty: Would anyone like a nice cup of Tea? An armchair guide

Eveline Baker – JEB Creative Solutions Pty Ltd

Brynn Hibbert – University of New South Wales (UNSW), Sydney

Method Validation: “Would a Vogon sign off on this Method?” The importance of fit for purpose methods

Presenter

Traceability: “I make it 42, you make it 42, but is it the same 42?” Routes to metrological traceability for field measurements

Lectures and presenters in the “Hitchhiker’s Guide to Quality Assurance”

Lecture

Table 1

648

649

Metrological traceability, the tracing of a measurement result to stated references (often a definition of a unit for a kind-of-quantity) might not be the only “traceability” required during a chemical measurement. On this theme Vanessa Hodgson from the Australian Federal Police explained how physical traceability of material secured from a crime scene must also be assured. The “chain of custody,” required to prove the integrity of the evidence, therefore starts when the evidence is first discovered and continues right up until the evidence is presented in court as part of the legal proceedings. This chain must include information on who has had contact with the evidence, at what time, under what circumstances, and what changes, if any, were made to the evidence. ISO/IEC 17025 [4] Sect. 5.8 refers to the handling of test and calibration items. However, for forensic purposes, the requirements are more stringent. In Australia, additional protocols have been specified by NATA [5]. It has been stressed in these pages, and as one of the six VAM principles, that competent people to carry out chemical analyses is of great importance. Pieter Scheelings opened his presentation with the quotes, “You are what you hire” and “Do not underestimate the ingenuity of complete fools.” The paper reviewed the roles of staff involved in quality assurance in a laboratory, from the CEO and quality manager downwards. The importance of having a quality philosophy from the highest levels in an organization was stressed, together with the people who have to translate that philosophy into action. Much of the advice given by Dr. Scheelings to managers recruiting staff who could be seen as likely ‘quality drivers’ is shown in Table 2. To conclude some of the characteristics of incompetent and disinterested staff were enumerated and ways of re-engaging them discussed. Dr. Baker has given courses on method validation throughout Australia, and her presentation was to set the scene for one of the workshops on the second day. As with many presentations, the requirements of ISO/IEC 17025 was a starting point, although simply complying with requirements was put in context of the underlying reasons for method validation. A laboratory needs to demonstrate its ability to use standard methods for which extensive method validation has been carried out and published (verification), but for all other cases some method validation needs to be done in house. This might range from a completely new method developed by the laboratory, for which a full suite of tests are required, to the use of standard methods outside their scope. So if a laboratory uses a standard method with a matrix not mentioned in the standard, or uses it outside the nominal concentration range, or uses non-specified equipment, then the ability of the laboratory to use the method with those changes must be demonstrated. Reference was made to both the EURACHEM guide to method validation [6] and the recent Harmonized guidelines from IUPAC [7]. The traditional aspects of method validation were explained with the modern additions of metrological traceability and measurement uncertainty. In the time for the presentation, it was not possible to cover all factors in detail, but many of the issues were discussed in the workshop the following day. (The Vogons of the title were particularly unpleasant

Table 2 Personal attributes that require close consideration in the selection of professional laboratory staff include the following∗ • Appropriate undergraduate study preferably covering all disciplines of chemistry • Academic excellence (indicates an achievement-orientated personality) • Postgraduate training or qualifications (indicates good problem-solving skills as well as professional discipline and perseverance) • Relevant experience (meets short-term organisational needs) • Involvement in sports or community activities (indicative of leadership, team-based skills and community spirit) • Excellence in sports, music, the arts or other non-science activities (indicative of creative and/or achievement-oriented personality) • Active involvement in professional bodies or staff associations (indicative of a positive professional attitude and good interpersonal and negotiation skills) • Postgraduate or further study in management disciplines (indicative of initiative and ambition for professional advancement) ∗

Reproduced from [13] with permission

and bureaucratic people who were actually responsible for the destruction of the Earth.) Adam Crawley presented the ‘top-down meets bottomup’ philosophy of the NMI in Australia. First, he felt a need to address the often heard denial that for whatever reason a particular laboratory does not need measurement uncertainty (no time, no data, measurement uncertainty is negligible compared with sampling, clients do not want/ do not understand measurement uncertainty, . . .). Like many accreditation bodies, NATA in Australia had given a period of time for laboratories to comply with the measurement uncertainty clauses of ISO/IEC 17025, but by now all laboratories are expected to demonstrate their assessment of measurement uncertainty. The controversy of bottom-up GUM versus the pure top down Horwitz approach was discussed [8] and the pragmatic synthesis, taking whatever route can do the job proposed. An example of fat in milk products was worked through in the following day’s workshop, using cause-and-effect diagrams to identify uncertainty components. Brynn Hibbert then gave a presentation on calibration as a stand in for an indisposed Mary Mulholland. In collaboration with Dr. Mulholland, he has often pointed out the problems that arise when the assumptions of a linear calibration fitted by a classical least squares procedure are violated (correctness of the linear model [9], all error in y which is random and homoscedastic). Also an anathema is the reliance on the correlation coefficient [10]. The lecture tied in with the earlier discussion of metrological traceability by highlighting the need to use traceable calibration materials. Independently prepared solutions and random order of analysis is also recommended. The virtues of oneand two-point calibrations in well validated and understood systems was also discussed. Uncertainties on fitted

650

parameters and the standard error of the regression are all available in Excel, and should be regularly computed. The standard deviation of an estimated concentration also highlights interesting aspects of calibration, why it is better to make measurements in the centre of a calibration range, why six points should be considered a minimum for a calibration and how replicated measurements reduce the uncertainty. In the discussion, Dr. Baker warned against forcing calibrations through zero, and was generally supported by the speaker and audience. The NMI is a major producer of reference materials for identification of drugs of abuse, used in sport and for other uses. Dr. Davies is the head of the section responsible for preparation and characterisation of pure reference materials. Many of the materials are deuterated to aid identification by mass spectrometry, and synthesised and characterised in house for identity and purity. The exercise was as much to justify the apparently high cost of such CRMs (apparently, because the real cost of preparing and characterizing by an NMI is much greater than the commercial price), as to explain what is being certified in a CRM. (A paper outlining the quality assurance for the NMI program for pure reference materials has been reported in this journal [11].) The use of CRMs in establishing metrological traceability and for other quality purposes were discussed, also strategies to be adopted when an appropriate CRM was not available. Ivan Waples, retired from NATA and long-time assessor, confronted the question of how to deal with an audit. Positing that they are perceived ‘a waste of time’, ‘bureaucratic nonsense’, ‘a real nuisance’, ‘fault-finding’, ‘stressful’ and so on, he made a good job of then presenting the positive side of an audit, whether internal (as required by ISO/IEC 17025) or external for accreditation. (Dr. Waples also used the nasty Vogons as a model. The invocation ‘Don’t Panic’ is written in large friendly letters on the cover of the eponymous Hitchhiker’s Guide). As with many aspects of quality assurance, the approach and mindset of the participants is all important, and if viewed correctly, an audit can be seen as an opportunity for improvement. ISO/IEC also complies with the appropriate 9000 series, which includes reference to quality improvement measures, including audits. He explained the process of an accreditation audit, the possible outcomes and how an organisation might react to this. Kirsty Outhred is the regional coordinator for chemical testing in NATA, and the timing was right for her to update the meeting on the revision to ISO/IEC 17025 published in 2005. The major change was to bring the standard into line with the revisions of ISO 9001:2000. Thus if a laboratory is accredited to ISO/IEC 17025 its quality management system will also meet the principles of ISO 9001. There were also some changes in terminology, in particular the ‘client’ of the old standard is now the ‘customer’ of the new. NATA is not requiring an immediate rewriting of SOPs for this change, which was some cause for relief, but the new terminology will need to be brought into updates and revisions as they are made. In Sect. 4, concerning management, a new Sect. (4.10) requires the

techniques in Sect. 4 to be applied to continuous improvement of the system effectiveness. The staff now have to be aware of the ‘relevance and importance’ of their activities, and how this is done has to be documented in the management system. Seeking feedback from customers is now required and this data must be analysed to improve customer services. In the technical section, staff training must now be assessed against objective and pre-defined criteria (5.2), as should laboratory performance and any corrective action taken (5.9). John Eames covered a lot of ground on quality control tools and approaches. Internal quality control requires analysis of different kinds of samples, some of known quantity value, including blanks (method, reagent, field), replicate samples, matrix spikes, surrogates, internal standards, calibration verification solutions, interference check standards, certified and in-house reference materials. For the latter, Dr. Eames detailed checks for homogeneity and stability. Dr. Eames has had extensive experience in elemental analysis of geological samples by ICP-MS and the concerns of this background come through in the lecture. Perhaps one of the difficulties faced by new quality managers is how to put together an effective mix of quality control tools that is cost-efficient and appropriate for their laboratories. Total Quality Management was introduced by the last speaker, Maree Stuart. She considered accreditation to ISO 9001 (or through ISO/IEC 17025), and different Awards: Baldridge (USA), Deming (Japan) and the Australian Business Excellence Framework and other approaches presently used in manufacturing. TQM is a strategic commitment to make quality and customer satisfaction a guiding factor in everything an organisation does. While this could be a guiding principle in a refrigerator factory, the approach needs some thought in chemical analysis. Customer satisfaction is a curious thing when it comes to chemical measurement results. Not all customers appreciate what makes a good result, and whether a result is cost effective and fit for purpose (or fit for intended use) might not be apparent until some time after the initial customer has received it. When we discuss quality in chemical laboratories, we often have in mind a super-customer (or perhaps meta-customer, or even virtual-customer) on which our quality attentions are focussed. TQM appears to be littered with bon mots or lists of principles, for example Deming’s 14 points, Deming’s “seven deadly diseases,” Crosby’s “four absolutes of quality,” the Joiner triangle, the seven step “plan-do, checkact” and so on. More recently these appear to have come together in the six-Sigma approach [12]. Some of the most useful tools to come out of TQM, that are useful in a laboratory, are the various mapping graphics such as flow charts, cause-and-effect diagrams, bar and pareto charts, histograms, control charts and scatter plots. Also from statistical process control comes concepts such as process capability index, and process performance index. The lecture ended with a comparison of the business awards and accreditation mentioned above.

651

Day 2 The second day revolved around workshops that built on principles outlined on day 1. There were worked examples of measurement uncertainty budgets, validation exercises, in-house production of reference materials, construction of control charts and approaches to proficiency testing schemes. Eveline Baker facilitated a hands-on session in which groups were asked to develop and validate methods to (1) measure the diameter of a planet from data of an occulting asteroid, and (2) measure the average water use during showering. The exercise was designed to cause the participants to think creatively about problems, not to over engineer solutions and to bear in mind intended use of the results. A real-life example was given of a pharmaceutical product that had to be recalled when a large number of customers reported worsening of symptoms on taking the drug. The material available could have consumed the day, and Dr. Baker cheerfully admitted she had used her two-day method validation course as a template. The workshop on preparation of in-house reference materials was more of a presentation that drew heavily on work by Brookman and Walker from the Laboratory of the Government Chemist. Reference materials produced in an organization are designed to eke out more expensive CRMs. To be of any use they must be properly characterized, be homogeneous, and be stable over the lifetime they are to be used. The presenters carefully explained the steps required for this production. Assessment of measurement uncertainty and final documentation and authorization were also stressed as important factors. Adam Cowley from the NMI used the case study of the estimation of measurement uncertainty for the analysis of cocaine base in drug preparations. Starting with the declaration of the measurand (the mass fraction of cocaine in a sample), and working through cause-and-effect diagrams a GUM bottom-up uncertainty budget was evolved. Precision data was used to draw together the Type A components and estimates of Type B were then included. The session ended with a fierce debate between Dr. Crawley and other presenters over the extent of alleged double counting of effects, but nevertheless it could be shown that the final estimate was reasonable and was consistent with different ‘reality checks’.

Dr. Kim Colyvas, who had not presented at the previous day’s session, gave a step-by-step guide to constructing and interpreting control charts, particularly CuSum charts. He showed that a cumulative sum chart (a time plot of the cumulative difference between the result of a measurement of a quantity embodied in a reference material and its assigned value) can be an effective visual tool, as well as the statistical uses to which it is put. By observing changes in the slope of the chart and matching these with events in the laboratory, important information may be obtained about the effect of day-to-day events. For example, if the column is changed on a chromatograph, or the usual analyst goes on leave, a CuSum chart of the analysis of a regular CRM picks up the effects of these. Dr. Kolyvas argued that often in a laboratory there is a step change that endures after the change in conditions (e.g. the new calibration solution has a different bias than the previous). This accumulates with time and the CuSum chart shows this up clearly. The day concluded with a round-table discussion on proficiency testing (PT). The first topic was to find out why laboratories participated in PT schemes. Was it just to comply with their conditions of accreditation? Some laboratories had joined schemes as part of their own QC program, and some used them as a method validation and precision study. There was some discussion about the interpretation of reports from PT schemes, and how laboratories react to their results. Conclusion The community of field laboratories in Australia have a desire to provide their customers with the best possible results. The lack of formal training in these aspects of analytical chemistry (there is only one course on QA/QC at the masters level in Australia) has meant that the Hitchhiker’s Guide to QA series of seminars, organized by the professional body, the Royal Australian Chemical Institute, have become well patronized events. It is important that field laboratories have many opportunities to learn of the advances in metrology in chemistry and approaches to quality assurance.

652

References 1. Hilton G, Comino E, Krieger B (eds) (2005) The Hitchhiker’s guide to quality assurance: the Queensland edition. The Royal Australian Chemical Institute, Brisbane 2. Hibbert DB (2006) Accreditation and quality assurance (in press) 3. De Bi`evre P (2006) Accreditation and quality assurance 10:519–520 4. ISO/IEC (1999) ISO/IEC:17025 General requirements for the competence of testing laboratories. International Organization for Standardization. Geneva, Switzerland

5. NATA (2002) NATA:ISO/IEC 17025 Application document: supplementary requirements for accreditation in the field of forensic science. National Association of Testing Authorities, Melbourne 6. EURACHEM (1998) Eurachem guide: the fitness for purpose of analytical methods. Laboratory of the Government Chemist, Teddington 7. Thompson M, Ellison S, Wood R (2002) Pure Appl Chem 74:835–855 8. Horwitz W (1998) J AOAC Int 785–794 9. Mulholland M, Hibbert DB (1997) J Chromatogr A:73–82

10. Hibbert DB (2005) Accred Qual Assur 10:300–301. DOI:10.1007/s00769-0050919-1 11. Hibbert DB, Westwood S (2003) Accred Qual Assur 8: Policies and concepts 12. i-Six-Sigma (2005) iSix Sigma Web site: http://www.isixsigma.com/ 13. Scheelings P (2005) Competent people – their impact and input in quality programs and measurement reliability. In: Hilton G, Comino E, Krieger B (eds) The hitchhiker’s guide to quality assurance: the Queensland edition. Royal Austral Chem Inst, Brisbane, pp 29–34