World J. Surg. 23, 1236 –1241, 1999

WORLD Journal of

SURGERY © 1999 by the Socie´te´ Internationale de Chirurgie

Development of Clinical Practice Guidelines: Surgical Perspective Melissa C. Brouwers, Ph.D., George P. Browman, M.D. Program in Evidence-based Care, Cancer Care Ontario, c/o Health Information Research Unit, Department of Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre, 1200 Main Street West, Room 2C9A, Hamilton, Ontario L8N 3Z5, Canada Abstract. Clinical practice guidelines (CPGs) are systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances. The implications of this definition are that: the methodologic perspective and operations for systematic development must be specified, a strategy is needed to account for the patient’s perspective in the CPG development process and the clinical decision, and a mechanism is required to determine how appropriateness ought to be conceptualized and defined. Addressing these issues, we review models of CPG development, outline challenges to evidence-based approaches to CPG development, address unique factors relevant to the development of guidelines for the surgical community, introduce an Ontario practice guidelines strategy that uses complementary methods of CPG development, and summarize the feedback provided by the surgical community regarding the practice guidelines produced in Ontario’s cancer system.

Clinical practice guidelines (CPGs) have the potential to facilitate decisions by clinicians and patients about appropriate health care [1]. Several approaches can be used to develop CPGs, including expert opinion, consensus methods (i.e., consensus among experts, consensus among various stakeholder groups), and evidence-based methods (i.e., restricted literature searches, systematic reviews) [2– 4]. Expert opinion, although less time-consuming and less expensive than the other approaches, is more vulnerable to bias and potential conflicts of interest. This perspective typically defines expertise by one’s content knowledge rather than by one’s experience and skill in research methodology, critical appraisal, and statistics as applied to the content area. These factors can lead to biased sampling methods, implicit consideration of the data, and unscientific procedures with which to resolve discrepancies in the interpretation and application of evidence [4]. As with expert opinion, various consensus processes have often been employed to create guidelines [4 –7]. The most reliable are those that include consideration of the scientific evidence, incorporate the views of various stakeholder groups, and employ methods to diffuse and control problematic intragroup interactions that influence recommendations (e.g., dominance) [7]. As with expert opinion, consensus approaches may engage in more implicit consideration of the evidence. In addition, there is the threat that the Correspondence to: G.P. Browman, M.D.

goal of reaching consensus may be pursued at the expense of appropriate use of the data. The evidence-based approach uses the results from health care research as the core information for informing clinical recommendations [4]. The major gain in moving from expert opinion or from consensus approaches alone toward an evidence-based approach is the explicitness in how recommendations are determined, which in turn renders evidence-based guidelines open to scrutiny. Evidence-based practice is defined as the conscientious, explicit, judicious use of the best available evidence from health care research for the management of individual patients [8]. Health care research may refer to external evidence from formal research studies (usually published), which can be generalized to a particular setting, or data generated from within the health care setting to which it will apply using the outcomes research approach. At the heart of the evidence-based approach using external evidence is the systematic review, which involves systematic and explicit processes for locating, selecting, assembling, synthesizing, analyzing, interpreting, and reporting the results of health care research [4, 8]. This process is intended to minimize bias in how research evidence is used, add to the reliability of evidence review, and make explicit the rules for applying the evidence in a manner that is open to careful scruntiny [9, 10]. Furthermore, this approach is less susceptible to problems of intragroup dynamics [7]. Accordingly, reviews of research should be as scientifically rigorous as the studies they review [8]. An evidence-based approach to CPG development is better able to assist in detecting the gaps between clinical research and clinical practice. This function is important, as previous research has demonstrated a strong negative relation between years since graduation and a provider’s knowledge of important advances in the detection, evaluation, and management of clinical problems, such as hypertension [11]. This relation is not surprising given that health care audiences may be unaware of existing research or may have difficulty accessing it; the sheer volume of research literature makes the task unmanageable [12, 13]. Indeed, there is currently estimated to be more than 250,000 published randomized controlled trials (RCTs) alone with an exponential increase in publication rate. Furthermore, it is inappropriate to assume that all audiences have the skills and resources with which to evaluate and synthesize the evidence on an ongoing basis. Evidence-based clinical practice guidelines provide access to a distilled synthesis of

Brouwers and Browman: Clinical Practice Guidelines

relevant information organized for decision-making purposes [14]. Although there are compelling arguments in favor of the use of systematic reviews, some caution is warranted. There is emerging empirical evidence demonstrating that the quality of studies can influence the size of observed treatment effects and that the published literature is inherently biased toward positive studies [15–21]. With respect to study quality, exaggerated effects of interventions have been demonstrated even for randomized controlled trials where double-blinding and concealment of patients to groups has not been done or reported [15–17]. Cook and her colleagues examined evidence from 127 trials and found that studies judged be of low quality exaggerated treatment effects by 33% [19]. With regard to publication bias, Dickerson and her colleagues examined 737 studies approved by two American institutional review boards and found there was a publication bias toward studies that yielded statistically significant results [20]. However, contrary to popular belief, the investigators found that the bias originated not from the journal editors but, rather, from the authors, who were less likely to submit manuscripts from studies in which no statistically significant results emerged. In addition, in a study comparing pairs of RCT reports published in German and English and matched for lead author, 62% of the English-language articles compared with only 35% of the German-language articles reported significant differences in favor of the treatment, although the quality of studies in both languages was judged to be the same [21]. Such biases suggest that clinical judgments based on the peer review literature only may be overly optimistic. These findings have led systematic reviewers to employ comprehensive strategies for locating unpublished trials, using abstracts of trials, incorporating non-English-language studies, and searching the gray literature [22]. Reviews that include this additional information (including what might be held by pharmaceutical companies) may provide more realistic estimates of the effects of interventions, but to gather this information is laborious, time-consuming, and expensive, rendering such measures impractical for meeting the expectations of many guideline development programs. Using Patient Inputs The definition of CPGs explicitly acknowledges the role of the patient perspective in clinical decision making [1]. Patients’ views are best represented by the patients themselves. Their involvement can occur at two levels. First, patients ought to be part of any formal CPG development strategy (either at the stage of evidence gathering and interpretation or at the stage of CPG approval). Second, any clinical decision must account for patients’ values and preferences for alternative treatment options, and a well worded CPG should respect the available choice. This has important implications for how we evaluate CPG programs [23]. Appropriateness of Care In general, the appropriateness of a clinical intervention is determined by a combination of prevailing conventional practices, the scientific evidence of benefit, the applicability of the evidence to the particular clinical setting in which the intervention will be used, and the acceptability of the trade-offs between projected

1237

benefits and harms from the perspective of the provider and the patient. In the absence of good scientific evidence, other factors become important, including the values of the patient, how risk is assessed, and properly elicited clinical experience. Appropriateness of care for specific procedures is usually determined formally using structured processes involving expert panels who consider all of the inputs implicitly [6]. Use of expert panels alone has been shown to produce results that in one study were unduly sensitive to panel composition and therefore potentially unreliable [24]. The more standardized and concrete are the criteria for appropriateness, the more reliable is the method likely to be. Special Considerations for Surgical CPGs Surgical Trials and Implications for CPGs For health care interventions, the RCT is accepted as the most sound methodologic design to test the relative effectiveness of alternative treatments (e.g., current practice versus new surgical procedure). By random assignment of participants to study conditions, the RCT affords the opportunity to follow patients prospectively in a standardized manner, causally infer the effects of the interventions, and more confidently rule out alternative explanations for the finding [25, 26]. Despite the strength of the design, RCTs do present challenges [25–27]. To conduct an RCT is notoriously time-consuming, expensive, and for some clinical conditions neither feasible nor ethical. Furthermore, debate exists regarding the extent to which findings from RCTs are generalizable and where the burden of proof for generalizing study results lies. These general barriers are common to all RCTs. The execution of surgical RCTs presents the investigator with a unique set of challenges [27–30]. For example, it is exceptionally difficult to standardize surgical procedures, as several factors contribute to variation, including a surgeon’s experience and skill and unanticipated but necessary modifications to the procedures to accommodate the unique needs of the clinical situation. Because it is impossible to ensure that both the patient and the clinician are blind to the surgical procedure, there are various threats to the validity of the outcome including the placebo effect (i.e., effects observed in a trial that are not due to the treatment or intervention but, rather, to participants’ expectations of the effects of treatment) and the Hawthorne effect (i.e., where the participants’ knowledge that they are part of a study influences results differentially). Finally, participant accrual to surgical RCTs may be more challenging than RCTs in other domains. At the conclusion of clinical trials on new and emerging drugs, for example, participants are able to access the drug the study found most effective; typically, participation does not threaten the range of treatment choices available to the individual subsequent to the trial. In contrast, surgical procedures are typically permanent, and, therefore, participants in these trials forgo the opportunity to access alternative, and potentially more effective, procedures after the study is completed. Given this limitation, patients may be less inclined to commit to participating in these sorts of trials. Despite these challenges faced by the surgical community, high quality relevant research evidence exists and should not preclude the development of evidence-based CPGs. In surgery, CPGs can help guide diagnostic test selection and interpretation, the selection of other preoperative procedures and assessments, the ap-

1238

propriate indications for alternative technical procedures, and adjunctive intraoperative and postoperative interventions. When it comes to advising on the actual performance of surgical technique, formal certified educational programs, professional monitoring, and standard settings may be more important than CPGs as guides to influence practice. CPGs ought to be considered only one of a set of complementary strategies to improve clinical outcomes of surgical care. Outcomes Research and Implications for CPGs Outcomes research may be useful for identifying inappropriate variations in practice where a high priority CPG exists and to evaluate the effects of CPGs on practice patterns over time. There may be strong implications, both positive and negative, for CPGs as they relate to outcomes research and, in particular, outcome patterns associated with small-area variations and with volumeoutcome effects [31–35]. Research on small-area variation is focused on monitoring patterns of care as a function of country, province or region, city, and hospital [31–33]. Technologic developments in the area of the electronic medical record may increase the ease and accuracy of small-area variation measurement and allow rapid feedback to practitioners to assist them in improving their practice. Small-area variation in the use of breast-conserving surgery (BCS) has been a common finding internationally, and several studies have investigated factors to explain the observed differences [36, 37]. For example, Iscoe et al. in 1994 found significant variations among Ontario hospitals in the use of BCSs [36]. Hospital characteristics (participation in a clinical trial, affiliation with medical school), institutional treatment options (presence of radiotherapy, presence of chemotherapy), patient characteristics (age at diagnosis), and geographic variables (population of town or city) failed to account for these differences. Grilli and Repetto in 1995 found that geographic variation in the implementation of BCSs in Lombardia, Italy was more dependent on provider attitudes in each region than on patient characteristics [37]. Research on the volume-outcome relation examines patterns of care as a function of the volume of the procedures performed [33–35]. An association has been well documented in the literature such that for some procedures, the volume or frequency with which they are conducted is positively associated with better clinical outcomes (e.g., lower mortality rates). Several explanations have been advanced to account for this relation. The “selective-referral hypothesis” suggests that volume is a function of outcome; higher volumes are a consequence of good outcomes. Specifically, increased referral patterns emerge with or without (e.g., reputation alone) explicit knowledge of outcome indicators. Alternatively, the “practice-makes-perfect” hypothesis suggests that outcome is a function of experience; good clinical outcomes emerge as a consequence of the number of procedures performed. Specifically, the more procedures one completes, the more proficient one becomes and the better the outcomes. Evidence for both hypotheses has been found, as has the finding that the overall patterns may be inflated because of insufficient consideration of co-variables (e.g., case mix, discharge policies) which may modulate the effect [34]. Clearly, the phenomenon is important, yet complex, and requires further study. From a positive perspective, CPGs can assist in reducing some of the variation, such as that found in these examples, by acting as

World J. Surg. Vol. 23, No. 12, December 1999

a tool to assist clinicians and patients in determining the most appropriate treatment option for a specific clinical circumstance. Furthermore, CPGs can provide the rationale to health planners and policymakers to ensure fair, equitable distribution of resources so recommendations can be implemented. From a negative perspective, there is the threat that CPGs could be used as a punitive measure against institutions, regions, or individual clinicians where patterns of practice demonstrate significant and unexplained variation from the CPG recommendation. For example, one of the limitations in the interpretation of small-area variation research is that each individual study measures only a fraction of the total number of factors that can contribute to the pattern. When competing explanations are explored by using covariates (e.g., institutional characteristics, provider characteristics, patient characteristics), the pattern of variation typically remains and may yield conclusions suggesting that the pattern is indicative of inappropriate or suboptimal care (or both). Considerable caution must be exercised with this kind of interpretation and the response to the interpretation. It would be inappropriate to abuse the CPG and have it dictate the clinical decision rather than have it serve as a tool to complement clinical judgment and patient preferences. To overcome this threat, it is critical that explanatory research studies be undertaken to explore more fully the hypothesis associated with small-area variations and volume– outcome relations. Cancer Care Ontario Practice Guideline Initiative The Organization The Cancer Care Ontario (CCO) Practice Guidelines Initiative (PGI) is a project within the newly formed CCO Program in Evidence-Based Care. The Initiative coordinates the development of evidence-based CPGs for specific cancer conditions related to interventions that include treatment, screening, diagnosis, and follow-up. The CPGs are produced by provincial teams, known as Disease Site Groups (DSGs), each composed of health professionals (e.g., oncologists, surgeons, nurses), community representatives (e.g., patients, family members), and researchers who work collaboratively to produce CPGs for a particular cancer disease site (e.g., lung cancer, breast cancer). Methodology The process in CPG development involves the following steps [38]. 1. A clinical problem is identified and defined. 2. High quality research studies are selected, reviewed, and their results are integrated. 3. A systematic review of the study results leads to the formulation of a draft practice guideline, known as an evidencebased recommendation report (EBR). 4. The EBR is distributed to the practicing community for feedback. 5. The feedback is integrated into the report. 6. A final CPG is approved by the Practice Guidelines Coordinating Committee and is disseminated to relevant targets. The methodology employed by the Initiative combines an evidence-based approach using systematic review with a consensus model approach [38]. The evidence-based approach ensures that

Brouwers and Browman: Clinical Practice Guidelines

1239

Table 1. Respondents’ ratings of the quality of the evidence-based recommendation reports as a function of EBR type. Practitioner feedback questionnaire items and EBR type Rationale for developing the EBR is clear Surgical EBRs Nonsurgical EBRs A guideline on this topic is useful to clinicians Surgical EBRs Nonsurgical EBRs Literature search is relevant and complete Surgical EBRs Nonsurgical EBRs Summary of evidence is acceptable Surgical EBRs Nonsurgical EBRs Agreement with EBR as stated Surgical EBRs Nonsurgical EBRs Agreement that EBR should serve as a guidelinea Surgical EBRs Nonsurgical EBRs

Strongly agree (%)

Agree (%)

50.0 33.8

46.3 62.2

48.9 38.5

Neither agree nor disagree (%)

Disagree (%)

Strongly disagree (%)

1.5 1.0

1.5 2.0

0.7 0

46.0 53.2

4.4 5.4

0 2.4

0.7 0.5

22.6 18.2

61.3 61.1

13.9 20.2

1.5 0.5

0.7 0

27.0 22.4

70.1 72.2

0.7 4.4

1.5 0.5

0.7 0.5

38.0 26.5

52.6 67.8

3.6 3.4

5.1 1.5

0.7 1.0

83.6 85.0

7.5 9.0

9.0 6.0

EBR: evidence-based recommendation. For the first six administrations of the questionnaire, the scale on this particular item was yes, no, and unsure rather than a 5-point agreement scale. For the purposes of the summary, the two versions of this item were combined by equating the responses yes with strongly agree and agree as the first scale level, unsure with neither agree nor disagree as the second scale level, and no with strongly disagree and disagree as the third scale level. a

the best available evidence is used in an unbiased explicit manner by the DSGs to make recommendations for patient care. There are two consensus approaches incorporated into the development of these CPGs. First, members of the DSGs, as they evaluate the evidence, must work together to reconcile differences in the interpretation of the evidence, judgments of its applicability, and assessments of trade-offs between alternative interventions in terms of benefits, harms, and costs. The second consensus approach, the Practitioner Feedback (PF) stage of the guideline development cycle, involves obtaining and incorporating the opinions of the clinical community in Ontario in a systematic fashion [39]. To this end, a nine-item PF assessment questionnaire of the EBR is distributed to a sample of the clinical community for whom the recommendations are considered to be relevant. The questionnaire assesses several dimensions, including the clarity of the rationale, the usefulness of the guideline, the clarity and completeness of the literature search on which the guideline is based, the acceptability of the summary of evidence, agreement with the evidence-based recommendation, agreement that the document should be approved as a practice guideline, and intentions to use the guideline in one’s own practice. A five-point Likert-type scale is used by the respondent to indicate his or her agreement with each item. The questionnaire responses and comments of respondents are analyzed and incorporated into the final approved guideline. Results An analysis has been conducted and published on the PF responses from the first 10 EBRs completed by the Initiative [39]. Here we provide an update report based on the feedback collected specifically from the surgical community on the 22 EBRs that have undergone the PF process since the Initiative’s inception to May 1998. Surgeons provided feedback on two EBRs dealing

Table 2. Respondents who would use an EBR in their practice if it became a clinical practice guideline as a function of EBR type. EBR type

Yes (%)

No (%)

Unsure (%)

Surgical EBRs Nonsurgical EBRs

80.9 90.3

10.6 3.4

8.5 6.2

EBR: evidence-based recommendation.

specifically with a surgical procedure: surgical management of early stage invasive breast cancer (stage I and II) [40] and management of ductal carcinoma in situ of the breast [41]. They also provided feedback on 20 EBRs focused on nonsurgical, but relevant, treatment options. A total of 137 responses were received for the surgical EBRs, and 205 responses were received for the nonsurgical EBRs. Table 1 provides the ratings by surgeons for the first six items of the PF questionnaire. As evident from Table 1, surgeons rated both types of EBRs favorably with more than 80% of the responses falling in the strongly agree and agree levels for each item. Table 2 outlines the intentions of the respondents from the surgical community to use the surgical and nonsurgical guidelines in practice. In summary, results from the PF analyses lend strong support from surgeons for the CPGs produced by the Initiative. To date, 17 CPGs have been completed and approved, with an additional 50 in development. Through a collaboration among the Program in Evidence-Based Care, the CCO Surgical Oncology Network, and the Institute for Clinical Evaluative Sciences (ICES), it is hoped that the special issues relating to evidence-based surgical oncology practice can be addressed in Ontario. Conclusions A successful CPG development strategy can influence clinical practice and lead to improved patient outcomes. Clinical decision

1240

making is a combination of: evidence; provider, patient, and systemic cultural values; and circumstances specific to the clinical and life situation of the patient and to the health care setting [42]. CPGs represent an opportunity to put our investment in health care research to proper use. Through systematic and explicit methods, we can provide and organize information about the best care as suggested by scientific evidence. A good guidelines development strategy uses evidence but also respects the inputs of other stakeholders, including patients, and the opinions of experts. However, the opinions of experts alone, undisciplined by a formal and explicit consideration of the evidence, should be considered with caution because of inherent biases that can affect how evidence is used and how opinions are expressed and adjudicated. For surgical practice, the evidence from outcomes research indicates that for some procedures experience is an important part of good practice [35], and CPG development for surgical interventions must consider how to deal with this issue. CPGs can help most where the experiential component of cancer deals with processes about the selection of patients and procedures, whereas other complementary strategies are needed to deal with the technical component of care that is so important for this discipline. At a minimum, guidelines for practice should be explicit about how the evidence is used and how other factors are considered. In the area of surgical practice, evidence from outcomes research seems especially relevant, and its proper interpretation requires attention if we are to respond appropriately. The surgical community must contribute to this area of scientific health care and continuous quality improvement. Re´sume´ Les «recommandations» de pratique clinique ou les «guidelines» sont des affirmations syste ´matiques pour assister le me´decin ou le patient `a prendre la meilleure de´cision pour une situation clinique pathologique spe´cifique donne ´e. Ceci implique une me´thodologie et des proce ´de´s ne´cessaires pour le de´veloppement syste ´matique des recommandations spe´cifiques, une strate ´gie pour tenir compte du point de vue du patient dans le de´veloppement des recommandations et dans la de´cision clinique, et un garde-fou pour assurer que le tout est cohe´rent, y compris sa conceptualisation et sa de ´finition. Pour re´pondre `a ces preble`mes, cet article: 1) revoit les mode `les de de´veloppement des recommandations et souligne les difficulte ´s souleve´es par une approche «factuelle» dans le de ´veloppement de ces recommandations; 2) traite les facteurs uniques ayant un rapport avec le de´veloppement des recommandations pour la communaute´ chirurgicale; 3) pre´sente la strate ´gie de recommandations adopte´e dans la province d’Ontario qui fait appel `a des me´thodes comple ´mentaires de de ´veloppement de recommandations, et 4) re´sume le retour d’information de la part de la communaute´ chirurgicale en ce qui concerne les recommandations dans la prise en charge des patients cance´reux dans la province d’Ontario. Resumen Las guı´as de pra´ctica clı´nica (GPC) son pronunciamientos desarrollados en forma sistema´tica para ayudar al me ´dico y al paciente en la toma de decisiones pertinentes al cuidado de la salud en circunstancias clı´nicas especı´ficas. Tal definicio ´n implica

World J. Surg. Vol. 23, No. 12, December 1999

que es necesario especificar la perspectiva metodolo ´gica y los procesos para el desarrollo sistema´tico, que se requiere una estrategia para responder a la perspectiva del paciente, tanto en el proceso de desarrollo de las GPC como en la decisio ´n clı´nica, y que se necesita un mecanismo para determinar co ´mo conceptualizar y definir lo que significa “apropiado.” Al discutir estos asuntos, el presente artı´culo: 1) revisa los modelos para el desarrollo de GPC y plantea los desafı´os pertinentes al enfoque basado en la evidencia para elaborar GPC; 2) enfoca aquellos factores especı´ficamente relevantes para desarrollar guı´as para la comunidad quiru ´rgica; 3) presenta la estrategia de las guı´as de pra´ctica clı´nica de Ontario que utiliza me´todos complementarios para el desarrollo de GPC y 4) resume la retroalimentacio ´n por parte de la comunidad quiru ´rgica en relacio ´n a las GPC producidas por el sistema de ca´ncer de Ontario. Acknowledgment Support for the Program in Evidence-based Care is provided by the Ontario Ministry of Health and Cancer Care, Ontario. References 1. Woolf, S.H.: Practice guidelines: a new reality in medicine. I. Recent developments. Arch. Intern. Med. 150:1811, 1990 2. Hayward, R.S.A., Laupacis, A.: Initiating, conducting and maintaining guidelines development programs. Can. Med. Assoc. J. 148:507, 1993 3. Woolf, S.H.: Practice guidelines, a new reality in medicine. Arch. Intern. Med. 152:946, 1992 4. Browman, G.P.: Evidence-based cancer care and clinical practice guidelines. In American Society of Clinical Oncology Educational Book: Thirty-fourth Annual Meeting, M.C. Perry, editor. Alexandria, VA, American Society of Clinical Oncology, 1998, pp. 451– 457 5. Kosecoff, J., Kanouse, D.E., Rogers, W.H., McCloskey, L., Winslow, C.M., Brook, R.H.: Effects of the National Institutes of Health Consensus Development Program on physician practice. J.A.M.A. 258: 2708, 1987 6. Brook, R.H., Chassin, M.R., Fink, A., Solomon, D.H., Kosecoff, J., Park, P.E.: A method for the detailed assessment of the appropriateness of medical technologies. Int. J. Technol. Assess. Health Care 2:53, 1986 7. Linstone, H.A., Turoff, M., editors: The Delphi Method: Techniques and Applications. Reading, MA, Addison-Wesley, 1975 8. Sackett, D.L., Rosenberg, W.M.C., Gray, J.A.M., Haynes, R.B., Richardson, W.S.: Evidence-based medicine: what it is and what it isn’t. B.M.J. 312:71, 1996 9. Mulrow, C.D., Cook, D.J., Davidoff, F.: Systematic reviews: critical links in the great chain of evidence. Ann. Intern. Med. 126:389, 1997 10. Cook, D.J., Mulrow, C.D., Haynes, R.B.: Systematic reviews: synthesis of best evidence for clinical decisions. Ann. Intern. Med. 126:376, 1997 11. Sackett, D.L., Haynes, R.B., Taylor, D.W., Gibson, E.S., Roberts, R.S., Johnson, A.L.: Clinical determinants of the decision to treat primary hypertension. Clin. Res. 24:648, 1977 12. Mulrow, C.D., Oxman, A.D., editors: Cochrane Collaboration Handbook [updated 9 December 1996]. Available in The Cochrane Library [database on disk and CDROM]. The Cochrane Collaboration, Issue 1. Oxford: Update Software, 1997 (updated quarterly) 13. Haynes, R.B.: How clinical journals could serve clinician readers better. In: The Future of Medical Journals: In Commemoration of the 150 Years of the British Medical Journal, S. Lock, editor. London, British Journal of London, 1991, pp. 116 –126 14. Williamson, J.W., German, P.S., Weiss, R., Skinner, E.A., Bowes, E.: Health science information management and continuing education of physicians: a survey of US primary care practitioners and their opinion leaders. Ann. Intern. Med. 110:151, 1989 15. Colditz, G.A., Miller, J.N., Mosteller, F.: How study design affects outcomes in comparisons of therapy. I. Medicine. Stat. Med. 8:441, 1989

Brouwers and Browman: Clinical Practice Guidelines 16. Chalmers, T.C., Celano, P., Sacks, H.S., Smith, H., Jr.: Bias in treatment assignment in controlled clinical trials. N. Engl. J. Med. 309: 1358, 1983 17. Schulz, K.F., Chamlers, I., Hayes, R.J., Alterman, D.G.: Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J.A.M.A. 273:408, 1995 18. Simes, R.J.: Publication bias: the case for an international registry of clinical trials. J. Clin. Oncol. 4:1529, 1986 19. Moher, D., Cook, D.J., Jadad, A.R., Tugwell, P., Moher, M., Jones, A., Pham, B., Klassen, T.P.: Assessing the quality of reports of randomized trials: implications for the conduct of meta-analyses. Health Technol. Assess. 3:1, 1999 20. Dickerson, K., Min, Y.I., Meinert, C.L.: Factors influencing publication of research results: follow up of applications submitted to two institutional review boards. J.A.M.A. 263:374, 1992 21. Egger, M., Zellweger-Zahner, T., Schneider, M., Junker, C., Lengeler, C., Antes, C.: Language bias in randomised controlled trials published in English and German. Lancet 350:326, 1997 22. Naylor, C.D.: Grey zones of clinical practice: some limits to evidencebased medicine. Lancet 345:840, 1995 23. Browman, G.P., Levine, M.N., Graham, I., Whelan, T., Sawka, C., Pritchard, K.I., Jadad, A., Newman, T.E.: Clinical practice guidelines: an evolving health care technology. Cancer Prev. Control 1:7, 1997 24. Leape, L.L., Park, R.E., Kahan, J.P., Brook, R.H.: Group judgments of appropriateness: the effect of panel composition. Qual. Assur. Health Care 4:151, 1992 25. Chalmers, T.C.: The clinical trial. Health Society 59:324, 1981 26. Pocock, S.J.: Current issues in the design and interpretation of clinical trials. B.M.J. 290:39, 1985 27. McLeod, R.S., Wright, J.G., Solomon, M.J., Hu, X., Walters, B.C., Lossing, A.: Randomized controlled trials in surgery: issues and problems. Surgery 119:483, 1996 28. Langfitt, T.W.: Are randomized clinical trials of surgical procedures feasible? Clin. Neursurg. 35:43, 1989 29. Chalmers, T.C.: Randomization of the first patient. Med. Clin. North Am. 59:1035, 1975 30. Solomon, M.J., McLeod, R.S.: Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 118: 459, 1995

1241 31. Health Services Research Group: Small-area variations: what they are and what do they mean? Can. Med. Assoc. J. 146:467, 1992 32. Choudhry, M.K., Wright, J.G., Singer, P.A.: Outcome rates for individual surgeons: concerns about accuracy, completeness, and consequence of disclosure. Surgery 115:406, 1994 33. Luft, H.S., Bunker, J.P., Enthoven, A.C.: Should operations be regionalized? The empirical relation between surgical volume and mortality. N. Engl. J. Med. 301:1365, 1979 34. Luft, H.S., Hunt, S.S., Maerki, S.C.: The volume-outcome relationship: practice-makes-perfect or selective-referral patterns. Health Serv. Res. 22:157, 1987 35. Sowden, A., Aletras, V., Place, M., Rice, N., Eastwood, A., Grilli, R., Ferguson, B., Posnett, J., Sheldon, T.: Volume of clinical activity in hospitals and healthcare outcomes, costs, and patient access. Qual. Health Care 6:109, 1997 36. Iscoe, N.A., Goel, V., Wu, K., Fehringer, G., Holowaty, E.J., Naylor, C.D.: Variation in breast cancer surgery in Ontario. Can. Med. Assoc. J. 150:345, 1994 37. Grilli, R., Repetto, F.: Variation in use of breast-conserving surgery in Lombardia, Italy. Int. J. Technol. Assess. Health Care 11:733, 1995 38. Browman, G.P., Levine, M.N., Mohide, E.A., Hayward, R.S.A., Pritchard, K.I., Gafni, A., Laupacis, A.: The practice guidelines development cycle: a conceptual tool for practice guidelines development and implementation. J. Clin. Oncol. 13:502, 1995 39. Browman, G.P., Newman, T., Mohide, E.A., Graham, I.D., Levine, M.N., Pritchard, K.I., Evans, W.K., Maroun, J.A., Hodson, I., Carey, M.S., Cowan, D.H.: Progress of clinical oncology guidelines development using the practice guidelines development cycle: the role of practitioner feedback. J. Clin. Oncol. 16:1226, 1998 40. Mirsky, D., O’Brien, E., McCready, D.R., Newman, T.E., Whelan, T.J., Levine, M.N., Provincial Breast Disease Site Group: Surgical management of early stage invasive breast cancer (stage I and II). Cancer Prev. Control 1:10, 1997 41. Wright, J., Whelan, T., McCready, D., O’Mailey, F., Provincial Disease Site Group: Management of ductal carcinoma in situ of the breast. Cancer Prev. Control 2:312, 1998 42. Sackett, D.L., Richardson, W.S., Rosenberg, W., Haynes, R.B.: Evidence-Based Medicine: How to Practice and Teach EBM. New York, Churchill Livingstone, 1997