THREE DILEMMAS IN THE INTEGRATED ASSESSMENT OF CLIMATE CHANGE

An Editorial Comment

Though current work in integrated assessment (IA) of global environmental change stands on foundations at least twenty years old, the level of interest and attention of the past five years is unique. Moreover, the past two years have shown increasing signs of maturation of the field - the first review articles and conferences, as well as a series of contributions summarizing cumulative results, critiquing current practice, and calling for development of a more connected professional community, with associated communication channels and professional standards. Two papers in this issue of Climatic Change make significant, though early, contributions, to these attempts to define and advance a discipline of integrated assessment (IA). Granger Morgan and Hadi Dowlatabadi present policy-relevant insights drawn from five years work in the Carnegie-Mellon IA project, and state their priorities for further work. James Risbey et al. add their voices to calls for professional standards, propose a taxonomy of forms of critical criteria, and critique a few systematic pitfalls of current practice. Morgan and Dowlatabadi summarize a set of contributions that are rich and astonishingly far-ranging, including the following: estimating costs and effects of electric utility demand-side management programs; estimating and valuing impacts of sea-level rise; modeling and valuing changes in terrestrial ecosystems; constructing dynamic indices to compare effects of multiple trace gases; estimating orderof-magnitude costs and effects of geoengineering measures; designing adaptive policies; measuring public knowledge and attitudes toward climate change and the environment; and estimating the effects of a carbon tax. As a separate contribution, they present a checklist of rules of good practice for doing IA. This sensible and wise list, presumably distilled from the experience of their project, charges the integrated assessor to focus on uncertainty; to iterate; not to neglect areas of ignorance; to include values explicitly, preferably parametrically; to assess the issue in broader social context; to span the problem through coordinated multiple analyses, not a single Procrustean model; and to support multiple assessments. This final item presumably addresses not assessors, but the sponsors and users of assessment. Risbey et al. do not survey the field, but stalk bigger, more dangerous prey defining and applying professional standards for IA. Their paper presents many opportunities for criticism and dissent, but they are above all to be commended for taking on such troubled and important questions. They propose a preliminary critique of IA, through sketching three classes of critical criteria: discipline-based Climatic Change 34: 315-326, 1996.

316

EDWARDA. PARSON

criteria to evaluate IA components; process-based criteria to evaluate the methods and processes of integration; and 'ends-based' criteria to address the use of IA in informing decision and policy. They defer discussion of the third class of criteria for a later paper, while providing general discussion of the second and detailed application of only the first. In addition, they sketch some proposals for process and practice to help effect incremental improvement in IA standards. These are both valuable contributions, that represent early progress in presenting cumulative knowledge and developing collective standards and fora. But it is early progress indeed, and a great deal of work remains to be done. Some basic problems of IA remain to be addressed, or even posed clearly. In this editorial, I summarize three fundamental dilemmas that confront Integrated Assessment. Each represents a challenge more basic and more difficult than any particular representational problem. Failure to manage these dilemmas will put IA at risk of foolish claims and excessive enthusiasm, and subsequent over-critical reaction. While none of these dilemmas admits simple solutions, understanding and managing them would support clearer thinking about potential contributions of assessment, reduce the risks of avoidable pitfalls, and promote the development of a professional community and critical standards that both contributions to this volume advocate.

Dilemma #1: Interdisciplinarity and Critical Standards The first dilemma stems from the broadly inter-disciplinary nature of IA, and concerns the relationship - intellectual, and managerial - between IA projects and the constituent disciplinary pieces on which they rest. Used for whatever purpose, IA supplements and depends on disciplinary inquiry, but does not supersede or replace it. The task of IA (as opposed to specific problems of understanding and representing behavior of component domains within an IA) is usefully to integrate and synthesize knowledge from disparate domains. This task poses various tensions between IA activity and its contributing disciplines, particularly as regards the definition and application of professional standards. The relationship between IA and contributing disciplines can fail by being either over-critical or under-critical. To illustrate the task of IA, and the relations and tensions between it and contributing disciplines, Risbey et al. have employed (as have others) the useful but imperfect image of bricks and mortar. (Another useful, often proposed image is of a jigsaw puzzle.) IA projects are regarded as structures assembled by skilled craftspeople from bricks of disciplinary knowledge, joined by mortar of models or other integrating methodologies.* Bricks arrive in an IA project, previously vetted by the relevant disciplinary standards. But bricks need mortar to join them, which must sometimes span a fair gap. Bricks may also have to be cut to fit, or may be at risk of failure when used other than their maker intended. Finding appropriate * These imagesecho Guetzkow'scharacterizationof formalmodelsin social science,as 'bridges between islandsof theory' (Guetzkow,1962).

EDITORIALCOMMENT

317

standards or review procedures in such a context is difficult. Disciplinary standards associated with component pieces of analysis may not suffice, may conflict with each other, or may fail to address crucial integrating components. Risbey et al.'s specific critiques of current IA practice illustrate this tension, particularly the risk of falling into either over-critical or under-critical stances. For instance, their criticism of the wholesale, uncritical importation of small chunks of analysis (which they call 'archetypes')* from their disciplinary origins into IA projects, charges the IA community with being under-critical of constituent pieces. They suggest that this uncritical adoption will be most grave from those 'feeder' disciplines with which IA practitioners are less expert. There is an important ambiguity in their argument here, though. It is not clear whether they mean the IA community is unable to apply the appropriate critical standards that prevailed in the home discipline, and so are too credulous with their imports, liable to misuse or over-extend them; or that these imports have not been regarded critically enough in their home disciplines, and that IA should recognize this disciplinary lapse and apply more demanding critical standards. While the prescriptions associated with these two interpretations would differ, the characterization of the risk in each case is similar. In each case, the use of archetypes is not problematic merely because weak results or analyses may be used in IA projects. The authors acknowledge that IA projects cannot treat every aspect in detail. Rather, to gain a comprehensive view or highlight broad links, IA must exclude, and must use simple schematic representations when richer ones are available. Indeed, the specific archetypes that Risbey et al. denounce can all be regarded as either parameters (for which you could substitute another value if you disagree), or simple place-holders (for which you could substitute a richer representation). The use of archetypes is problematic, though, if repeated uncritical use of a simple place-holder, or narrow arguments over a parameter value, constrain argument and so prevent people from noticing that the parameter, or the placeholder, may be fundamentally misconceived. (This could arise through simple complacency, or through community pressure to obtain 'a number' to meet a modeling need.) Arguments over whether AEEI is zero, one, or two per cent per annum are not fruitful if the concept of AEEI is incoherent; analogous arguments apply to exponential discounting and equilibrium aggregate numerical climatedamage functions. So whether these archetypes are harmful will depend on whether and how they constrain IA debate. This is an empirical question, which in my view is still unresolved. For each of Risbey et al.'s three examples, social pressure to provide single numbers to plug into models presently appears to co-exist with heated critique of the underlying concepts. This may well represent a fruitful and healthy tension. * Their examples include Nordhaus' estimate of economic damage from an equilibriumdoubledCO2 climate, autonomous energy-efficiencyimprovement (AEEI) indeces, and conventional exponential discounting.

318

EDWARDA. PARSON

Risbey et al.'s other specific criticisms of current practice illustrate the same tensions. They identify the IPCC 'value-of-statistical-life' controversy as an instance of the use of policy-analytic tools outside their appropriate domain. More broadly, they imply that when IA'ers have only partial familiarity with the origin and limits of methods they use, they are systematically at risk of such inappropriate extensions. Similarly, their discussion of the TARGETS group's audacious attempt to employ Cultural Theory in IA criticizes them for employing a set of concepts whose merit and limits they were unable to judge. But the authors also proceed to judge Cultural Theory themselves, both by listing a collection of social phenomena that it excludes (and presumably should include), and by blithely denying one of its core assertions.* The use of Cultural Theory in IA poses the problem of appropriate critical relations between IA and its contributing disciplines very sharply. Cultural Theory, while contentious in the extreme in its home disciplines, is predictably attractive to heroic IA modelers. Its bold claims to universality (it is probably the only social theory claiming universality since the decline of Talcott Parsons' sociology), and the plausibility with which it can be interpreted as making precise predictive claims reducible to formal representation, both make it so. Who then is in a position to evaluate its application in integrated assessment? Its home disciplines (sociology and anthropology) have no remotely unified view of it; and IA'ers, with limited connections to these disciplines, are unlikely even to understand the controversies. While the application of Cultural Theory to IA maybe a hard case to which no plausible answer is available, a few insights into how to evaluate IA projects (and how not to) are available from careful reflection on its bridging, integrating status, supplementing and depending on, but not superseding or replacing, disciplinary inquiry. First, since IA's tasks are synthesis of knowledge across domains, and its constructive application to public deliberation and policy choice, IA cannot be held liable for the general incompleteness of relevant knowledge. If the standard for useful IA is specific, spatially and temporally precise, authoritative predictions of global environmental change, with all associated human drivers and impacts if imprecision and indeterminacy of knowledge imply inadequacy of assessment then all IA'ers should give up now. More gently, when basic understanding of relevant causal processes is lacking, IA's are not likely to create it. They can help - they may reveal or characterize the need, motivate the research effort, perhaps even provide hints about fruitful ways to proceed - but they cannot do the job. The standard visual images illustrate both IA's potential contribution, and its limits: you must try to assemble the jigsaw puzzle to see which pieces are missing; mortar can fill a small hole, but cannot replace whole missing bricks. Indeed, when large pieces of constituent knowledge are missing, the best contribution of IA may be to identify what research is most needed to illuminate the -

* 'Uncertainties in [biophysical] parameters are . . . very weak functions of world view' (Risbey et al., 1996, p. 382).

EDITORIALCOMMENT

319

whole system, and seek to make contributions there. The CIAP project of the 1970s, for example, the first major integrated environmental assessment and the largest to date, spent much of its resources commissioning disciplinary research for which it had identified the need (Grobecker, 1974; Glantz et al., 1985). On climate change, an IA project that is candid about gaps in underlying knowledge may sometimes best contribute by trying to stimulate - or by doing - work that looks more like disciplinary contributions than integrated assessment. Indeed, this describes many of the results summarized by Morgan and Dowlatabadi. This strategy will yield a selection of projects that may appear arbitrary or eccentric, will inevitably be incomplete, and will certainly not look integrated. But a problem as broad as climate change is hard to span, and seeking to span the entire scope from the outset is clearly not the only useful approach. Indeed, when gaps in underlying knowledge are too severe, IA projects that appear to integrate broadly and seamlessly may only do so by suppressing uncertainty, or by delimiting the problem to exclude what does not fit. When IA stimulates or undertakes component research, the primary contribution of the integrated perspective is a stance from which to identify what is needed. But this is a risky path, since it will often lead to IA'ers becoming interlopers in related disciplines. Whether an interloper can do good research in a neighbor discipline and whether a neighbor discipline can assimilate and benefit from good work by an interloper, particularly when the work was motivated by priorities of synthesis or policy-relevance that came from outside the discipline - are open questions, and there are IA'ers who carry the scars to prove it. Second, IA's bridging status also has implications for evaluating projects that do attempt to span many domains. Practitioners of IA cannot be required to be more brilliant than everybody else, nor masters of every domain that falls under their inquiries. Hence, the critical standards applied to IA cannot be the intersection of all potentially relevant standards from constituent domains. To do so would apply standards that may not be appropriate, and reduce all evaluation of the endeavor to cheap-shots. But saying what standards should not be, does not take us very far. If not these standards, then what? Only partial, suggestive, and procedural answers are available. First, for IA projects that adopt the strategy of identifying and pursuing important gaps in component knowledge, the answer is relatively easy. Standards and review procedures from the relevant disciplines should normally prevail - in the hope that few members of the discipline are so outraged by the invasion that they shoot first and ask questions later. For this kind of work, Risbey et al. are on target when they argue that disciplines should set standards of adequacy, while the integrating agenda, and its potential users, should set standards of value. When IA projects pursue broader integration, standards can only emerge from processes of negotiation and mutual persuasion involving both within-discipline and across-discipline reviewers. Participants in this process must seek in good faith to advance standards of both integrating work and cross-disciplinary conversation,

320

EDWARD A. PARSON

and agreeing to disagree can only be a provisional, temporary resolution. Necessary for such progress, but far from sufficient, will be process criteria such as Risbey et al. state - e.g., clear documentation of assumptions (including those embedded from inherited or imported components), and vigilant guarding against anchoring or other unrecognized biases. The place they propose for 'dirty laundry' is a fine idea, if incentives can be designed to support honest disclosure of errors, pitfalls, and unsuccessful attempts. Finally, as Morgan and Dowlatabadi's checklist suggests, iteration - within IA projects, and between IA and both constituent disciplines and the policy world - will be essential to the development of standards and review procedures.

Dilemma #2: Historical Prediction vs. Exhortation

The second dilemma stems from the long time-horizons and global scales that IA must encompass. Because IAs examine climate change, impacts, and consequences over periods of many decades, they often require projections of human choice and behavior, and their environmental consequences, over such periods. How will fertility rates change, globally and regionally, over the next century? If real incomes in many regions continue to grow sharply, what will this mean for the energy and material intensity of consumption patterns? Will the character of aggregate technological change over the next century increase or decrease energy and carbon intensities? And how, and how much, can any of these changes be influenced by policy? IA's dependence on such projections poses problems of representation more severe, perhaps of fundamentally different character, than those stemming from interdisciplinarity. On these questions, there may be fundamental limits to the kind of understanding that permits predictive, causal representation and modeling. We do not have a general, causal, predictive theory of history, and neither building models, nor including sociologists, anthropologists, historians, and philosophers on assessment teams - though these are both good and important things to do will give us one. IA's very strengths make it prone to error in this domain. Where knowledge is lacking, IA can reveal and characterize knowledge needs, guide inquiry, and help pose focused questions. But particularly for formal IA modeling, which encourages thinking about whole systems in terms of 'wiring diagrams' of causal relations, being able to pose focused questions puts you perpetually at risk of finding (or being offered) answers to them, and believing them - even for questions that cannot be answered, now or perhaps ever. IA projects have taken three approaches to managing this dilemma of projection. The first, and most widespread, has been to create multiple, alternative 'scenarios': packages of related projections, typically of regional and global population and economic growth, and relevant technological change, which are presumed to span,

EDITORIALCOMMENT

321

or to illustrate, the range of plausible futures.* This approach, of which the IPCC Scenarios are the best-known example, is well accepted as a pragmatic starting point, but has several well-known problems: the projected ranges tend to reflect subtle biases toward continuance of recent trends, and hence too narrowly frame a 'status quo' future; simultaneous projections of multiple quantities may embody undetected contradiction or incoherence, or unstated presumptions regarding active policy intervention; and most fundamentally, such projections cannot adequately reflect the possibility of surprise or discontinuity, which may dominate historical change. Several projects seeking to craft 'surprise-rich' scenarios have demonstrated the vast breadth of plausible futures, but contributed little to the problem of creating positive planning scenarios. Two alternative approaches can partly avoid requiring assessors to predict the future. The first would make assessments explicitly normative, so the projections they necessarily embody represent either desirable futures to be pursued, or undesirable ones to be avoided. Such assessments can support questions such as 'what would it take to bring us here', or 'what are the salient risks pushing us there'; their projections are used not as prediction or illustration, but as exhortation. The second alternative approach would seek to make assessment tools so flexible that projections and planning scenarios can be specified by various users. None of these approaches fully avoids the problem of prediction, though each has some merit, as do hybrid approaches. This dilemma, while distinct from the problem of policy advice, can only be managed in the same context.

Dilemma #3: Useful Advice to Policy The third dilemma concerns IA's aspiration to bridge expertise and politics, to provide useful advice to help inform political, policy, and management decisions. For this often-cited central purpose of IA, the two dilemmas stated above operate at full force, compounded by further problems that characterize the fuzzy border between the domains of politics and expertise. IA can (or should) seek to bring relevant expert knowledge to bear on political decisions, but many other factors must also inform such decisions: preferences and values, resources, ideologies, institutional arrangements and possibilities, precedent, connections with other issues, etc. In seeking to inform policy, IA activity is, as Risbey et al. contend, subject to evaluation by additional criteria that are not relevant when IA is regarded only as an interdisciplinary research activity, derived from and reflecting back on its constituent research domains. At least two additional kinds of critical criteria may apply: principled standards of fair process, participation, and legitimacy; and (for want of a better term) strategic criteria that hold assessment responsible for its foreseeable use (and misuse) by partisan actors in a pluralistic political setting. * Documentation of scenarios normally stresses that they only illustrate plausible future trends, but in subsequent use they are normally treated as exhaustingthe relevant page.

322

EDWARD A. PARSON

These multiple kinds of evaluative criteria do not combine easily. As I argued above for combining disciplinary standards, simultaneous application of all relevant standards is not a workable solution, for any assessment endeavor will be killed by a thousand arrows. Adding still more critical standards only shoots more arrows into the corpse. But how, then, can we recognize assessment that contributes more constructively or usefully to policy debate, when expert knowledge is not authoritative, and political values are diverse, obscure, contested? And by what standards, even approximate or pragmatic ones, should it be judged? In practical terms, these deep problems appear in several specific, related forms, bearing on the linked questions of what kind of analysis is done for assessment, and how assessment results and insights are presented, and perceived, in policy debates. For example, any IA project must decide the scope of phenomena it seeks to represent (e.g., human activities, technologies, emissions, environmental characteristics, impacts, responses, and valuation). In so choosing, assessors face a strict trade-off between defending the scientific authoritativeness of their work, and attaining policy relevance. Policy-relevance is advanced when IA answers instrumental questions about the connections between policy choices and valued consequences. But providing such answers requires spanning domains of knowledge that differ markedly in the confidence and precision with which things are known, and the degree of consensus among experts. In policy debate, such broad synthesis is likely to be only as strong as its weakest link. To protect assessment from partisan deconstruction and critique, assessors have often narrowed the scope of their work, restricting it to domains where precise statements of knowledge and strong expert consensus are attainable, sacrificing policy relevance to defend the authoritativeness of a core of expert advice (Jaeger, 1996; Clark, 1996). Whether this strategy advances the usefulness of assessment is not clear: by making certain narrowly stated propositions secure from partisan attack, it may under some conditions delimit policy debate and facilitate agreement; but it renounces the possibility of expert input to any questions except those on which their unity and authority are absolutely secure. A second example concerns how the meaning of assessment results is presented and used in policy debates. Here, Risbey et al. (after Wynne and Shackley, 1994) use the 'truth machine' vs. 'heuristic' dichotomy: assessments as statements or predictions, however uncertain, about the nature of the world; or assessments as instructive illustration of possibilities, embodying non-predictive 'insights' about possible choices, risks, contingencies, and linkages. Risbey et al. insightfully point out the artificiality of this distinction, because the non-predictive 'insights' that assessments claim to offer are themselves, or depend on, predictions: contingent, appropriately qualified predictions that are presumed robust to some level of underlying uncertainty, but predictions nonetheless. What are the implications of this dependence of qualitative insights on quantitative predictions? In part, as Risbey et al. argue, it merely increases assessors' responsibility to be as clear as they can about uncertainties, assumptions, and

EDITORIALCOMMENT

323

embedded values. Of course, if values and assumptions are so weakly perceived or deeply embedded that they cannot be identified and rooted out on command, then this exhortation will be ineffective and other approaches may be needed. But a more fundamental problem concerns the role and limits of uncertain, contested, or 'illustrative' analysis in informing policy debates, and how (if at all) such non-authoritative knowledge can best contribute. There is much persuasive force to the rationalist intuition that even uncertain or partial scientific knowledge should be able to help inform public choice. But the manifest weakness of such results makes them highly vulnerable to various forms of partisan exploitation and attack, as well as sincere misinterpretation. Many forms of error and misuse are common: clearly stated uncertainties may be taken to favor inaction; widespread misinterpretation of an assessment statement may accidentally favor one side or another in a complex polarized debate; identification of multiple possibilities may confuse policy debate or unintendedly increase, or decrease, political force for action; careful updating of estimates over time may yield changed numerical estimates or projections that are taken as capriciousness, and the assessment community lose credibility. These processes may be at odds with enlightened policy discourse and decision - and are certainly painful for politically innocent assessors and analysts. But all the obvious simple antidotes seem clearly wrong: not doing assessment at all; contracting its scope to those answers that will 'stand up in court', and hence abandoning policy relevance; waiting until broad policy choices have been made, and confining assessment to ways of realizing and implementing them; or seeking to preempt political decision-making, by biasing presentation of assessment results to favor the policy choices that assessors themselves have decided are necessary or desirable. But if these are the (widely practiced) wrong answers, what are the right ones? How should IA's be done, and presented and used in policy debate, to help advance the debates, and protect against the most extreme forms of (or consequences of) either sincere misunderstanding or partisan abuse? As on the other two dilemmas, only partial, suggestive, prescriptions are available. I sketch two directions here. One essential part of the answer concerns better use and presentation of uncertainties in policy domains. By this, I do not mean merely better education of policy actors in the scientific use and meaning of uncertainty. Rather, the concept of uncertainty itself must be more closely connected to corresponding concepts of variation as they operate in policy realms. We need better ways of connecting uncertain physical quantities first with the domain of expert dissent (i.e., progress on the problem of aggregating or pooling expert judgments); and better ways of connecting these physical and expert processes with analogous concepts that operate in the policy realm, including variation of effects (who gains, who loses, how badly hurt are the worst 10%, etc.), and variation of preferences and values. Somehow, we must move toward a common construction of uncertainty that consistently integrates these now disparate, and often confused, concepts.

324

EDWARD A. PARSON

While I have sketched this agenda only very roughly, two broad results summarized in Morgan and Dowlatabadi suggest precisely the kind of progress in integrating these two domains that is needed. These two results, which frame their paper, combine assumptions about physical uncertainty and inter-personal variation in preferences, to articulate in-principle limits to the contribution of formal analyses to identifying preferred policies. First, in early work with a very simple assessment model, they show that different preferred policy choices were determined far more strongly by different values than by physical uncertainties; hence, reductions in parametric uncertainties were unlikely to attenuate policy conflict. In later work with a much more sophisticated model, they find that under uncertain variation of both parameters and model structure, no policy choice attains stochastic dominance for all world regions. Together, these results provide strong suggestive support for the limits of formal analysis in resolving climate-change as a policy problem. In analytic terms, these results suggest a focus on multi-actor representations including multiple valued outcomes. More broadly, this agenda suggests that there may be great value in finding innovative processes and methods to link assessment to policy debate - increased involvement of diverse policy actors in the development and use of assessments and assessment tools; increased export of flexible assessment tools into policy realms, for various actors to exercise and modify according to their purposes; and increased development of experimental devices such as simulations, scenario-planning exercises, and policy exercises, to bring diverse policy expertise together with assessors and their models and tools to explore the limits, interactions, and borders of their respective domains.

Conclusions These three dilemmas embody the hardest, most important, and most enduring problems of doing assessment well. None admits simple, obvious solutions. Each can be managed better or worse for any particular assessment endeavor, but doing better requires clear understanding of the purpose of the endeavor. What ways of combining different pieces of disciplinary knowledge, of making projections, and of pursuing policy relevance are more or less appropriate will differ, depending on whether a project seeks to characterize uncertainties and gaps in knowledge; to advise a particular policy choice; to support dialog among policy actors; or to facilitate inquiry into relevant values or goals. Evaluation of the relative emphasis, the methods, and the process of an assessment can only be done relative to some such purpose. Of course, some pitfalls may be so serious as to thwart any purpose, as Risbey et al.'s discussion of the global modeling movement reminds us. The global models' most obvious pitfalls - inadequate treatment of uncertainty, neglect of economic adjustment, excessive confidence in predictions - have largely been seen

EDITORIAL COMMENT

325

and avoided by the current assessment community (though there may be more to be learned even here). But on the subtler questions of how assessment or modeling can contribute most usefully to policy, little progress has been made since the 1970s. Consequently, though assessment has advanced in many ways since then, IA remains at risk of suffering the same fate as the global models: a cycle of early enthusiasm, followed by a reaction of frustration and excessive, undeserved rejection. Current endeavors in IA have made substantial contributions to identifying and prioritizing knowledge needs, less to informing specific policy choice. Further progress cannot be guided by a single canonical view of what assessment should be and do, but will proceed incrementally down multiple paths. Several paths currently appear promising: analytic approaches to better represent multiple actors, diverse preferences, and multiple valued outcomes; better representation and application of uncertainty, including diverse expert opinion; novel methods to link assessment with policy communities; and broader participation in assessment teams and explicit focus on negotiating and elaborating pragmatic, viable critical standards. Risbey et al.'s call to develop institutions for critical reflection, mutual learning, and self-improvement will be crucial in developing and evaluating the progress made down these paths. Morgan and Dowlatabadi's checklist for desiderata of IA is a good starting point for a conversation about assessment standards, to which I would propose a few extensions and elaborations. First, there should be not just multiple assessments, but multiple assessment projects using diverse collections of methods and approaches. Second, assessment projects should explore novel methods for connecting their work with the policy community. Third, the approach should be iterative not just within each project, but across assessment projects and between them and the policy community. Fourth, assessors should not be embarrassed by, or seek to disguise, results that are merely illustrative, non-authoritative, and suggestive; these should be acknowledged as such, and the vigorous questioning and critique that will come, including partisan critique, accepted. Do not seek to avoid criticism by mumbling. An important limit to this checklist approach is suggested, though, by the way various writers have groped to define assessment standards by analogy to other domains, revealing how limited is our understanding of how to evaluate assessment. Risbey et al. refer to 'connoisseurship', as if assessment is like fine wine; Clark and Majone (1985) refer to artistic criticism, as if assessment is like opera singing. If these analogies are appropriate, then pursuing a single set of critical standards for assessment is at least premature, possibly erroneous. Rather, there should be a diversity of approaches, perhaps so broad that no single set of criteria for excellence could be defined. The pragmatic middle way between the too-limiting application of a single set of standards, and an anarchic refusal to evaluate, will have to be negotiated, defined, and improved incrementally.

326

EDWARD A. PARSON

References Clark, William C. and Majone, Giandomenico: 1985, 'The Critical Use of Scientific Inquiries with Policy Implications', Science, Technology, and Human Values 10 (3), 6-19. Glantz, Michael H., Robinson, Jennifer, and Krenz, Maria E.: 1985, 'Recent Assessments', in Kates, Robert W., Ausubel, Jesse H., and Berberian, Mimi (eds.), Climate Impact Assessment: Studies of the Interaction of Climate and Society, SCOPE 27, International Council of Scientific Unions., Wiley, Chichester. Grobecker, A. J., Coroniti, S. C., and Cannon, Jr., R. H.: 1974, The Report of Findings: The Effects of Stratospheric Pollution by Aircraft, DOT-TST-75-50, US Department of Transportation, Climatic Impact Assessment Program, National Technical Information Service, Springfield, VA. Guetzkow, Harold (ed.): 1962, Simulation in Social Science: Readings, Prentice-Hall, Englewood Cliffs, NJ. Morgan, M. G. and Dowlatabadi, H.: 1996, 'Learning from Integrated Assessment of Climate Change', Climatic Change 34, 3--4. Risbey, J., Kandlikar, M., and Patwardhan, A.: 1996, 'Assessing Integrated Assessments', Climatic Change 34, 3-4. Wynne, Brian and Shackley, Simon: 1995, 'Environmental Models - Truth Machines or Social Heuristics', The Globe 21, 6--8. John F. Kennedy School of Government, Harvard University, 79 JFK Street, Cambridge, MA 02138, U.S.A.

EDWARD A. PARSON