Accred Qual Assur DOI 10.1007/s00769-015-1143-2
PRACTITIONER’S REPORT
Application of indicators and quality index as a tool for critical analysis and continuous improvement of laboratories accredited against ISO/IEC 17025 Raquel Helena Catini1 • Fernando Jose´ Pires de Souza1 • Maria de Fa´tima Martins Pinhel1 • Andre´ de Oliveira Mendonc¸a1 Vitor Hugo Polise´l Pacces2 • Igor Renato Bertoni Olivares2
•
Received: 8 January 2015 / Accepted: 26 May 2015 Springer-Verlag Berlin Heidelberg 2015
Abstract In this work, a model for developing indicators and quality index was developed and applied, meaning to evaluate laboratories which have implemented the quality management system in accordance with the ISO/IEC 17025 requirements. Firstly, the chosen analysis tool (such as ‘‘brainstorming’’ and PDCA) to create a development of indicator and quality index model is discussed. This model was applied at the National Agricultural Research Laboratory of Sao Paulo—Laborato´rio Nacional Agropecua´rio do estado de Sa˜o Paulo (Lanagro-SP). As a result, a set of indicators was developed and applied which were grouped to obtain an indicator of unique quality for ten different laboratory units evaluated. This proved to be an essential tool for critical analysis and continual improvement of the quality management system in laboratories exercising the ISO/IEC 17025. Keywords Indicator Index Quality Continuous improvement ISO/IEC 17025
Electronic supplementary material The online version of this article (doi:10.1007/s00769-015-1143-2) contains supplementary material, which is available to authorized users. & Igor Renato Bertoni Olivares igorolivares@iqsc.usp.br 1
National Laboratory of Agriculture, Livestock and Food Supply (LANAGRO-SP), Raul Ferrari Street, PO Box: 5538, Campinas, Sa˜o Paulo 13100-105, Brazil
2
University of Sa˜o Paulo – Chemistry Institute (IQSC-USP), Trabalhador Sa˜ocarlense Avenue, 400, PO Box: 780, Sa˜o Carlos, Sa˜o Paulo 13560-970, Brazil
Introduction The implementation of quality management systems for laboratories in accordance with the ISO/IEC 17025 requirements [1] is a necessity in many areas, such as forensic laboratories [2], chemical pollution in surface water monitoring [3, 4], residues and contaminants in food monitoring [5] and sports doping control [6]. It is actually evident that ISO/IEC 17025 is established for analytical quality assurance, as it is required internationally, recognized worldwide and applied by different routine laboratories [7]. In Brazil, the Ministry of Agriculture, Livestock and Supply (MAPA), published the Normative Instruction 57/2013 [8] establishing new criteria for accreditation, recognition, scope expansion and monitoring of laboratories, in order to integrate the National Network of Agricultural Laboratories of the Unified System of Agricultural Health. As the basis for the accreditation process, this normative instruction depends upon being accredited by CGCRE/INMETRO (National Institute of Metrology, Quality and Technology—Brazilian accrediting entity which enjoys mutual recognition agreements with the full members of ILAC—International Laboratory Accreditation Cooperation) in conformity with the ISO/IEC 17025:2005 [1] standards. Thus, MAPA’s National Agricultural Research Laboratory of Sao Paulo (Lanagro-SP) is accredited since 2009. The ISO/IEC 17025:2005 requirements embody all the ISO 9001 requirements pertinent to laboratories [1], thereby embodying the process approach [9]. In every process, it is also possible to apply the continuous improvement cycle method (PDCA) [10] developed by Shewhart and refined by Deming. The PDCA cycle is designed to plan something, defining its objectives and processes in accordance with predetermined requirements
123
Accred Qual Assur
(P—Plan), execute the process (D—Do), verify through monitoring and measurement of processes and products to predetermined requirements—having these results disclosed—(C—Check), and finally, take actions to correct and improve the processes involved (A—Act). Seeing that ISO/IEC 17025:2005 is structured according to the PDCA model, it is possible to assess that it is necessary to pursuit continuous improvement after its implantation, understanding that among its various standard items, Requirement 4.15 (management reviews) is a key aspect. This requirement seeks to widely evaluate different system information, such as audit results, proficiency tests and corrective actions, among others. This information must be properly evaluated so that appropriate actions are taken to evolve (improve) the system. Along with the many different ways to evaluate those requirements, the application of indicators (a tool to obtain information about a particular reality) and quality index (aiming at describing more complex phenomena) can be considered a powerful mechanism, indispensable to evaluate a laboratory’s performance in its various processes [11]. Highlighting the importance of ISO/IEC 17025:2005 as a quality management system for laboratories and the need for appropriate processes of critical analysis, the objective of this article is development and application of a model on the basis of quality indicators for laboratories, with the intention of contributing to this purpose.
•
Participants are instructed to provide a large volume of ideas in a short period of time; Participants must ‘‘follow along’’ after different ideas, expanding, modifying or producing others by association.
•
Following the brainstorm sessions, the indicators were defined and decided how their effect could be rated; they were grouped to form a single number which would be the quality index, as developed by Santos [13] for environmental laboratories. In Fig. 1, a system is shown how to develop the indicators and quality index, structured within the PDCA concept. Details of each step are given below: 1.
2.
Approach To develop this work, at the outset, a process for development of indicators and quality index was elaborated and applied at Lanagro/SP to assess ten different laboratory units.
3.
Development of indicators and quality index model Choosing the type of indicator, whichever it may be, was made based on the particular needs of the laboratory. Brainstorming sessions were held with the departments’ leaders to define the indicators, their evaluation parameters and, subsequently, their assigned degree of importance. According to Juran [12], brainstorming is a group technique to generate constructive and creative ideas from all participants, following these rules: •
•
Good ideas are not supported immediately, as focus must be on developing new ideas rather than judging them; Ideas must be creative opposite to conventional;
123
4.
5.
Define group responsible for developing indicators In this stage, a person or a group of people (coordinators) was selected. We found it paramount that this person or group grasped good people skills and knowledge both on quality and conflict management. Literature review of relevant indicators Potential indicators were evaluated, such as Balanced Scorecard [14] (which considers customers’ innovation and financial perspective); Harrison and Meng’s proposal [14] (considering the cost of total quality); Takashina and Flores’s proposal [14] who use the National Quality Award model as reference (considering different criteria such as strategic plans and leadership, people, processes, customers, the society, outcomes, etc.); or still, Bezerra and collaborators’ proposal [11] (who consider the ISO/IEC 17025:2005 requirements as criteria to develop indicators, such as the amount of complaints, average time of deployment of corrective action, number of preventive actions, proficiency testing results and total number of errors in records). Laboratory quality management system study The quality management system was evaluated, being it already to the knowledge of the group coordinator, which gave prominence to management review that covers the system’s key points. Selection of experts Another group of people (experts) was chosen, who contributed and/or were interested in the indicator and index results, such as unit heads, supervisors, internal quality auditors and analysts. Planning brainstorming sessions 1 The coordinators selected potential indicators, which had already been elected by the specialists during brainstorming session 1. Also, an initial proposal and presentation with the work’s objectives was prepared to explain each indicator and the reason they were chosen.
Accred Qual Assur Fig. 1 Proposed steps to develop indicators and quality index
6.
7.
Brainstorming session 1 for indicator selection The brainstorming session was begun by introducing all indicators, previously selected by the coordinators and specialists, in an attempt to balance the level of knowledge. The session was concluded with the election. During the election (which was performed using a table to sum each vote for each indicator), it was proved important to emphasize the need of finding an optimal number able to adequately represent the reality of laboratories, minding not to have an excessive number which would hinder the analysis and processes. Planning brainstorming session 2 Considering the selected indicators, the coordinators then assigned a weight of significance to each indicator, based on the importance of that indicator. The table was organized and discussed with the group of experts during brainstorming session 2 (Table 1).
8.
Brainstorming session 2 to assign weights to indicators To conclude the table, brainstorming session 2 was held with the group of coordinators and specialists to debate Table 1, making it possible to insert new criteria for each indicator or modify the weights assigned to them. When no general consensus was reached, a voting session was held. To form the index (which corresponds to the sum of the scores of each indicator), the total weight summation of the indicators never exceeded 100. 9. Application of indicators and index shaping Laboratories were assessed through the indicators, assigning them a final score, so the index was formed (corresponding to the sum of the scores of each indicator). A table containing the indicators, criteria and weights was used. See example in Table 1. 10. Critical analysis of results The indicators and index obtained were periodically
123
Accred Qual Assur Table 1 Example of indicator assessment criteria Assessment parameter
a
1. Assessment of interlaboratory performance (last 6 months)
2. No. of non-conformities in external/ internal audits 3. Corrective measures within deadline 4. No. of improvements implemented 5. No. of complaints received
a
Results
b
Table 2 Developed indicators Weight
c
Score
d
Satisfactory in every occasion
5
Warning in some occasion
2
3. No. of internal training
Unsatisfactory in some occasion
0
4. No. of nonconformities identified during Audits (Internal and External)
0–5
5
6–12
2
\12
0
All
5
More than 70 %
2
\70 %
0
More than 3 1–3
5 2
None
0
None
5
1–2
2
More than 2
0
b
Add all assessment criteria for each indicator in ‘‘Results.’’ Designate a score of importance for each criterion c Assign a weight of significance to each criterion d
When development of indicator data collection is done, the result must be scored in this column, based on the weight of each criterion
reviewed through meetings with the experts when it was assessed whether changing indicators or their weights, selecting other experts, or further adjustments had to be made. Improvement actions Based on the critical analysis results, possible changes in the development of indicators and index formation were proposed and assessed.
Thus, the process of developing indicators and quality index was structured within the PDCA concept. Figure 1 features the planning stage (Plan), containing steps 1–8, including, within the steps, brainstorming sessions. After the planning stage was completed, data collection for the development of indicators and index was performed in step 9 (Do). Aiming to check whether the indicators and index were adequately portraying the reality of laboratories, step 10 allowed holding periodic meetings with the experts responsible for supporting the development of indicator process (Check). The indicators, their weights or any other factor were allowed to be modified based on the results of the meetings to adjust to the laboratory’s current reality (Act), continuously improving indicators and quality index. It is important to stress that getting employees to
123
1. No. of quality management system document (referred to documents revised within deadline) 2. Implementation of corrective actions
Enter all selected indicators in ‘‘Assessment parameter’’
11.
Indicators
5. No. of internal nonconformities identified routinely by the analysts 6. No. of preventive actions 7. Quality assurance tools 8. No. of external training 9. Analyzed samples 10. No. of internal audits 11. No. of improvements 12. No. of scientific production and knowledge transfer 13. No. of accredited scopes 14. No. of relevant complaints
participate during the selection of indicators was an essential factor as they were more committed when implementing them. Application of the model to develop indicators and quality index A model based on the development of indicators and quality index already presented was applied in 10 Lanagro/ SP laboratory units. At first, a series of meetings was held with the laboratories’ coordinators from the units evaluated. Most of these meetings were in the form of brainstorming sessions to define a criterion to select and score the indicators. Ultimately, it was possible to develop 14 quality indicators based on the ISO/IEC 17025 requirements, as shown in Table 2. Considering the indicators developed, it was important to detach the use of indicators linked to the number of nonconformities, because it was necessary to take into account the sampling nature of the assessment (i.e., the more assessments carried, the more findings, and it might not be a deterioration of laboratory performance). This way, if the internal or external audits program changed, the situation needed to be evaluated because it could influence the comparison of these indicators during different periods. To evaluate the indicators, a descriptive table with the scoring criteria was developed (available as Electronic Supplementary Material). This table was then given to the laboratories coordinators who were asked to fill them in (review of the first part—self evaluation). In some cases,
Accred Qual Assur
Fig. 2 Temporal development of the quality index of ten laboratory units (numbers on the top of the bars are quality index for each laboratories in different periods)
the indicator was not applicable at the period evaluated; therefore, a revised weight column was inserted in the table, where 0 was added for those cases. This was an important procedure for the total sum which constitutes the index. Therefore, the index was calculated by adding up all figures in the ‘‘revised weight’’ column, dividing it by the sum obtained in the ‘‘score’’ column and multiplying this ratio by 100 to obtain scores on a scale between 0 and 100. In this same table, an automatic graphic ‘‘speedometer’’ was inserted, capable of graphically displaying how well a unit performed in each indicator. After each laboratory unit’s self-assessment (by filling in the table), it was possible to designate a score for each indicator and quality index. Figure 2 shows the index for the different laboratory units (represented by ‘‘laboratory unit 1’’—‘‘laboratory unit 10’’) for 2 years (collecting information to evaluate the indicators was performed for each period of 6 months, so Fig. 2 shows four different scores representing each laboratory unit index variation in a period of 2 years). Figure 3 shows a compilation of the indicators assessed in all units in a way to check which indicators are better served by the units.
Conclusions The application of indicators and quality index stands out as a powerful tool for continuous improvement. It can also contribute to requirement 4.15 of ISO/IEC 17025 (management review). Besides its importance in measuring the quality management system performance, giving employee feedback must also be emphasized. The management staff can disclose the indicators and index onto a full view notice board, for example. The feedback is intended to engage and direct employees to continuously improve their exertion, striving for quality management system excellence. Regarding the evaluation of the laboratory units, the results were also considered very satisfactory. Each unit can occasionally check their indicators in a unique way, based on the ‘‘speedometer’’ (as showed in Electronic Supplementary Material) and also verify, in a broader way, its evolution opposite the ISO/IEC 17025 implementation and compliance, when comparing its index in different periods (Fig. 2). It is also possible to assess, in a global way (and as an excellent laboratory critical analysis tool) aside from visually assessing all units and their evolution (evaluating
123
Accred Qual Assur
Fig. 3 Compilation of indicators assessed in all units (average of indicators compliance)
Fig. 2), the percentage of criteria compliance through indicators (evaluating Fig. 3), making it viable to elaborate specific training courses, infrastructure, etc., in pursuance of improving such indicators, thus continuously evolve and develop the quality management system. Acknowledgments The authors acknowledge the CNPQ for a scholarship support programme; and the National Laboratory of Agriculture, Livestock and Food Supply by funds and facilities.
References 1. ISO/IEC 17025 (2005) General requirements for the competence of testing and calibration laboratories. International Organization for Standardization (ISO), Geneva 2. Malkoc E, Neuteboom W (2007) Forensic Sci Int 167:121 3. Lepom P, Brown B, Hanke G, Loos R, Quevauviller Ph, Wollgast J (2009) J Chromatogr A 1216:302 4. Commission European (2009) Commission Directive 2009/90/EC of 31 July 2009 laying down, pursuant to directive 2000/60/EC of the European parliament and of the council, technical specifications for chemical analysis and monitoring of water status. Off J Eur Union L201:36
123
5. European Commission, Commission Decision of 12 August 2002 (2002/657/CE) (2002) Implementing council directive 96/23/EC concerning the performance of analytical methods and the interpretation of results. Off J Eur Union L221:9–36 6. Chung KH, Choi GS, Lee W, Cho YH, Lee CW (2006) Accredit Qual Assur 10:603 7. Olivares IRB, Lopes FA (2012) Essential steps to providing reliable results using the analytical quality assurance cycle. Trends Anal Chem 35:109–121 8. BRAZIL (2013) Ministry of Agriculture, Livestock and Supply. Normative instruction no. 57 of December 11 9. ISO 9001:2008 (2008) Quality management systems—requirements. ISO, Geneva 10. Gupta P (2006) Quality progress magazine, v.39, n.7. Translated by Maricy de Queiroz Andrade and published in Banas Quality BQ_172/15 9/2006 11. Bezerra IL, Carreira W, Oliveira RM (2011) Indicators of performance for metrology labs. Bannas Quality Magazine 5:24–29 12. Juran JM (1999) Juran’s quality handbook, 5th edn. McGrawHill, Maidenherd 13. Santos CR (2009) Laboratory Quality Index (LQI): a proposal for environmental control laboratories. Doctoral Thesis. University of Sa˜o Paulo—School of Public Health 14. Martins RA, Neto PLOC (1998) Performance indicators for total quality management: a proposal of systematization. Manag Prod Mag 05:298–311
