Journal of the Operational Research Society (2011) 62, 1391 --1402

© 2011

Operational Research Society Ltd. All rights reserved. 0160-5682/11 www.palgrave-journals.com/jors/

How to use a systems diagram to analyse and structure complex problems for policy issue papers TE van der Lei, B Enserink∗ , WAH Thissen and G Bekebrede Delft University of Technology, Delft, the Netherlands Many policy problems are complex in the sense that natural, technological, social and human elements interact. Problem exploration and structuring are essential as a basis for deliberate and focused approaches towards problem resolution. The results of problem exploration efforts can be laid down in the form of a policy issue paper. We have developed a systemic, stepwise approach, which has been elaborated and taught for over a decade to hundreds of students. This seven-step approach centers on the construction of a system diagram as a means to provide structure to the conceptualisation of a complex problem situation. The approach is based on a conscious combination of existing relatively straightforward analytical methods including objectives hierarchy, means-ends analysis, causal diagramming, stakeholder analysis, and contextual scenarios. The obtained insights are then summed up in a policy issue paper, which is the basis for further planning and research. Journal of the Operational Research Society (2011) 62, 1391 – 1402. doi:10.1057/jors.2010.28 Published online 28 April 2010 Keywords: OR education; problem structuring; policy issue paper; policy analysis; system diagram; complex problems

Introduction Many policy problems are wicked, ill-structured or complex in the sense that natural, technological, social, and human elements interact. As a result, a variety of problem perceptions exists, values and interests may be conflicting, and power and resources to change things are distributed over multiple actors (Rittel and Webber, 1973, Dunn, 1994; de Bruijn and Porter, 2004; Koppenjan and Klijn, 2004). Such complexity is the everyday reality of analysts and problem solvers concerned with such complex socio-technological systems. Exploration of the problem situation and identification and selection of the key issues to be addressed are the first step that must be taken toward tackling the situation. Dunn (1994, p 106) explains this in the following way: Whereas well-structured problems permit analysts to use conventional methods to resolve clearly formulated or selfevident problems, ill-structured problems demand that the analyst first take an active part in defining the nature of the problem itself.

∗ Correspondence: B Enserink, Delft University of Technology, PO Box 5015, 2600GA Delft, the Netherlands. E-mail: [email protected] The first three authors work at the Policy Analysis section, the fourth at the Policy, Organisation, Law & Gaming section of the Faculty of Technology, Policy and Management of Delft University of Technology, the Netherlands.

In such situations analysts are advised to seriously invest in problem formulation activities before jumping to solutions, to prevent the risk of problems half-solved, growing worse or even solving the wrong problem (Quade, 1980). This structuring of an unstructured problem and defining an appropriate scope for further analysis and action are essential skills that all policy analysts should master. Therefore, when developing a novel curriculum in Systems Engineering, Policy Analysis and Management (see, eg, Thissen, 2000a), we were faced with the challenge to teach our (undergraduate) students a way of thinking supported by an analytic approach that would prepare them for their later work as practicing analysts. As a starting point, we noted that various authors have proposed to document the findings of problem exploration efforts in a so-called ‘policy issue paper’. Preparing it should force the analyst to address all relevant aspects of a problem situation in an analytically sound way (Ackoff, 1978; Miser and Quade, 1988; Quade, 1989). The policy issue paper helps to reach a joint problem understanding between analyst and client and it proposes a contribution to solving that problem by providing focus and proposing specific further steps. A number of authors have formulated requirements of a policy issue paper (Quade, 1980; Dunn, 1994; Checkland, 1985). All these authors indicate that a broad approach should be taken, looking at the situation from a variety of angles. However, these authors provide little guidance on how exactly to proceed to structure a problem for a policy issue paper. Although Dunn (1994), Quade (1980), and Checkland (1985) provide scarce guidance on how to structure a

1392

Journal of the Operational Research Society Vol. 62, No. 7

problem for an issue paper there is a complementary body of knowledge, called problem structuring methods (PSMs) that focuses on the art and craft of problem structuring. In a recent special issue of the Journal of the Operational Research Society, Shaw et al (2006) while referring to Rosenhead (1989) and Rosenhead and Mingers (2001) define PMSs as a collection of participatory modelling approaches that aim to support a diverse collection of actors in addressing a problematic situation of shared concern. The methods, generally, are participative in character and help to structure a problem together with the clients. Therefore adequate facilitation of the group process is crucial for success. An overview of different types of PSMs is given amongst others by Eden and Ackermann (1998) and Checkland and Poulter (2006). Three PSMs are dominant: Soft Systems Methodology, Strategic Options Development and Analysis, and Strategic Choice (Rosenhead and Mingers, 2001). Eden and Ackermann (2006, p 766) argue that the majority of the users of these methods tend to pragmatically combine parts of each of these three methods, with little regard for their theoretical underpinnings. In addition, they note that PMSs are not just used to structure problems, but rather to seek to facilitate agreements to act. As a result of the complexity of the ‘craft skills demanding many different roles for the ORer’, the approaches have become complex, less transparent than would be desirable, and difficult to transfer. The approach presented in this paper is different from the development of PSM’s in two respects. First, a policy issue paper is usually written by an individual analyst (or a small team) for a client or decision maker and is not typically the outcome of a participatory process. Rather, it is a problem exploration that might precede such a participatory process. The purpose of an issue paper is to explore and define a problem in sufficient depth so that a client may decide to do nothing, commission a study, proceed with a participative process in which the identified stakeholders will be involved, or combinations of these (Dunn, 1994, p 362; Checkland, 1985, p 168). Second, our approach is based on using and combining analytical concepts rooted in the ‘hard’ systems traditions (Sage, 1992; Sterman, 2000; Walker, 2000; Sage and Olson, 2001). This systems view focuses on analytical rigour, consistency and conceptual clarity rather than on the facilitation of stakeholder interaction. The approach has been developed in a period of more than 10 years of teaching to over 800 policy analysis students. The courses focus on teaching students how to structure and analyse a complex problem situation and how to write a clear policy issue paper based on this analysis. This is part of the core undergraduate curriculum in Systems Engineering, Policy Analysis and Management (see, eg Thissen (2000a) for more information on the curriculum). An earlier version of the approach has been presented at an IEEE conference (Thissen et al, 2000b). The steps that outline the approach will be illustrated by a simplified virtual case. By presenting this simplified stepwise methodological approach we intend

to contribute to transparency, as was urged by Eden and Ackermann (2006) as we want students and colleague practitioners to understand what analytical and modelling steps may be taken to construct a systems diagram; thereby inevitably compromising on the level of detail of describing the method.

The system diagram as a means to represent a structured problem We use the so-called system diagram (Sage, 1992; Sterman, 2000; Walker, 2000; Sage and Olson, 2001) as a core concept to represent a structured view of a problem situation. All of the analytical tools that will be presented as part of our approach have a relationship to the system diagram. The system diagram distinguishes the system itself, the steering factors, the external factors, and the outcomes of interest or criteria (see Figure 1). As we will illustrate in the next sections, the complete diagram is constructed through seven steps and iterations, where each step is a sub-analysis. Although our steps are presented as a neat sequence, in practice there is no perfect order and a lot of iteration.

A step-wise approach for constructing a systems diagram On the basis of the demands put forward in literature (Quade, 1980; Dunn, 1994; Checkland, 1985) we outline the questions a policy issue paper should address in Table 1. The first four questions help to structure the problem situation from the perspective of the problem owner. The first question, treated in the next section is intended to determine the problem level or scope from the perspective of the problem owner as suggested by Dunn and Checkland (based on Quade, 1982, pp 71–76), by demanding insight into the source and background of the problem and reasons for attention (Dunn, 1994, p 363; Checkland, 1985, p 169). The subsequent questions address the problem owner’s objectives, means and the system itself. The goals and objectives and measures of effectiveness are for example demanded by Checkland (1985). Also Dunn mentions the elements of goals and objectives, measures of effectiveness and potential solutions as part of an issue paper. Questions five and six relate directly to the environment or context of the problem of the problem owner, and help to assess the extent to which the problem owner depends on other actors and external factors beyond his control. Dunn

External factors

Steering factors

Criteria System of interest

Figure 1

The basic system diagram.

TE van der Lei et al—Using a systems diagram to analyse and structure complex problems

Table 1

1393

Questions that a policy issue paper should address

Question

Main focus

1. 2. 3. 4.

To structure the problem and provide focus from the perspective of a problem owner

Why should the problem be solved? What determines success for the problem owner? What are the means available to the problem owner? What is the system of interest?

5. Who else is involved? 6. How might the problem change in future?

To asses how the environment influences the problem of the problem owner and the possibilities to resolve it

7. What further steps, including possible research, are recommended?

To recommend future steps for the problem owner

adds an element of ‘major stakeholders’ and according to Checkland ‘beneficiaries and losers’ have to be described. The question about environmental changes is less central in the outline of an issue paper as proposed by Checkland and Dunn. However, the environment of ill-structured problems is dynamic and therefore we take it into account in the analysis. Question seven, is about what we learned from the analysis, assesses the information gathered and helps to draw conclusions and formulate recommendations for further research. This is comparable with Checkland (1985, p 169) ‘framework for analysis’ and ‘recommendations that may emerge.’ Both Dunn and Checkland go a step further in their elements in the issue paper by giving recommendations for solutions. Our approach ends with supported knowledge gaps, research questions, and a proposal for follow-on steps. These seven questions are the backbone of the stepwise approach that we have developed and used in our teaching efforts. For each question different analysis methods are used. We emphasize iteration, that is, re-assessment and re-formulation of the problem after each step. To illustrate the use of the analysis methods that belong to each question a fictional example is provided for an energy company called LightOn suffering from capacity shortage. We assumed LightOn has a need to reduce capacity shortage while taking costs into account. The example serves to illustrate the principles only and is not intended as a representation of a complete detailed structuring of a complex empirical problem. We used a limited set of mostly internet sources for constructing the case (Goodell, 2001; van Walle, 2006) and reports by Harrison (2004) and Smaardijk et al (2005).

between the present or expected future reality, and the desires or norms of the problem owner(s). It is of crucial importance to have a clear understanding of what really matters to the problem owner. Often, problems are formulated in terms of quite operational needs (such as the need to reduce the capacity shortage in our example). As a result, one may get ‘trapped’ into looking into a very narrow spectrum of solutions. Therefore, first of all, the more fundamental interests at stake should be explored. To this end, we use a ‘problem level determination’ diagram, which is based on the same logic as Keeney’s means-ends objectives hierarchy (Keeney, 1992), but concentrates purely on the ‘Why- question’; why would the problem owner want this problem to be solved? The diagram is constructed by starting from the initial formulation of needs, by searching for the underlying reasons for that need. Each time, the question is asked ‘Why is it important to satisfy this need or to solve the problem as initially articulated?’ This way, the more fundamental interests are identified. On the basis of the insights thus gained, a decision must be made regarding the level that is used as a starting point for further analysis and exploration of solution options. Choosing a more fundamental objective will generally open up a broader spectrum of solution options. The downside is that broader analyses will be more encompassing and challenging, or even infeasible if a very broad and fundamental objective is chosen, such as ‘to improve everybody’s happiness’. Note that any change in the choice of fundamental objective will imply a change in the initial problem formulation. Figure 2 below illustrates how such ‘problem level determination’ diagram may look like for our ‘LightOn’ example.

Question 1: Why should the problem be solved? Most policy decisions concern the choice of measures, the creation or modification of facilities or projects that are driven by a desire to solve problems, or at the very least to make these problems controllable. Before making such decisions, it is important to clearly describe what is considered to be the problem. As a working definition we use: ‘a policy problem is the gap between an existing or expected situation and a desired situation (a principle, or norm)’ (Hoogerwerf, 1988). In other words, a policy problem is the perceived difference

Question 2: What determines success for the problem owner? Once a leading objective is chosen, criteria are specified that can be used to ‘measure’ the extent to which the objective is attained. To this end, a so-called (fundamental) objectives hierarchy or objectives tree is used (eg Keeney, 1988, 1992; Dunn, 1994). The primary objective is split out in intermediate level objectives, and these are further specified until they can be expressed in measurable indicators. The latter are

1394

Journal of the Operational Research Society Vol. 62, No. 7

To provide longterm company continuity

Continuity of the company

To increase net profits

To increase turnover and income

Boundary

Good image

To reduce costs

High profit

To reduce capacity shortage

Figure 2 ‘Problem level determination’ diagram for LightOn.

High client satisfaction

More sustainable image

Figure 4 Example of an objectives hierarchy for LightOn.

Figure 3 The objectives hierarchy and its relationship to the right side of the system diagram.

called criteria and put at the right side or output side of the system diagram. Objectives are found by asking the ‘What is-question’; what attributes define this objective? A visualisation of the relationship of the objectives hierarchy and the system diagram is provided in Figure 3. It is important that the criteria do not overlap, that is, each must measure another aspect of what is desired (Keeney and Gregory, 2005). Objectives in the tree need to have a direction like more profit, less costs, less CO2 pollution, more recycling etc.

Question 3: What are the means or tactics available to the problem owner? How can the problem owner solve the problem; what are the means he can use to solve it and/or to make it smaller? Ideas for means or measures can be generated in different ways: through analysis of the means of the problem owner and other parties, by asking experts, by creative techniques like brainstorming, by reasoning backwards from the chosen objectives or by using checklists. Means or tactics can be of different nature (see also Walker, 1988): for example communicative (launch an awareness campaign); regulatory (create new environmental legislation); financial (provide subsidies) or technical (build a new power plant). We suggest to combine

the various methods and explore a broad range of possible means. We illustrate the use of a means-ends diagram or meansends objectives hierarchy (Keeney, 1992) by reasoning backwards from the objectives stemming from the previously developed objectives. The question to be answered is: How (by using which means) can the objective(s) be achieved? By systematically reasoning backwards from a chosen objective a (broad) spectrum of means is mapped out that may contribute to realisation of the objective (Figure 4). The means-ends diagram is based on assumed causality. Moreover means are associated with actions; therefore in the means-ends diagram we use verbs (see Figure 5). In principle the means can be interlinked (feed into one another) and feedback loops can occur. Implementation of the lowest level means affects the value of factors that influence the system and its outcomes. These factors are represented as steering factors on the left side of the system diagram. A visual example of the relationship of the means-ends analysis and the system diagram is provided in Figure 6. In Figure 6 the means are represented as blocks and the objectives as black dots.

Question 4: What is the system of interest? The system of interest is further structured and specified with the help of a causal diagram or causal model (Simon, 1952; Axelrod, 1976; Eden, 1988, 2004; Sterman, 2000; Bryson et al 2004). The causal diagram, as we use it, consists of factors and the influence relationships between these factors. The relationships can either be positive, depicted with a ‘+’, or negative, depicted with a ‘−’. A positive influence implies that if factor A increases, factor B will also increase (supposing

TE van der Lei et al—Using a systems diagram to analyse and structure complex problems

A influences B). A negative influence relationship means that an increase of A will lead to a decrease of B. If the character of the influence is unknown or cannot be determined this is indicated by a question mark. The causal diagram portrays the causal mechanisms linking the tactics with the criteria. These mechanisms are derived from established theories, definitions, expert knowledge, and/or beliefs of the author(s) of such a diagram. The intermediate factors represent key attributes of the system, and are portrayed inside the system box in the diagram. Besides factors that can be influenced by the problem owner, other factors may affect the relevant system outcomes, for example the weather. These factors may be identified by asking what other factors may influence the relevant factors in the system. As a result, we distinguish four classes of factors: (1) the steering factors related to the tactics at the left side of the diagram; (2) the criteria at the right of the diagram; (3) the intermediary or internal factors inside the box; and (4) the external factors that do influence the intermediary factors but cannot be controlled by the problem owner, which are located at the top-side of the diagram.

Continuity of the company

More sustainable image

High profit

1395

A visual of the relationship of the causal diagram and the system diagram is provided in Figure 7. An elaboration for the LightOn example is given in Figure 8.

Question 5: Who else is involved and what are their means to affect the interests of the problem owner? The analysis of the perspective and interests of the problem owner suffices when he himself has sufficient means to solve his problem adequately. But in modern networked societies most problem owners can only achieve their objectives in cooperation with others and by preventing strong opposition. Therefore the analyst should gain insight into what other parties will or should be engaged, what their interests are, what relevant means they possess, how they see the situation, and what their intentions are. A stakeholder/actor analysis and basic network analysis serve to identify relevant social, institutional, and political attributes of the problem situation. An example of the possible lay-out of the actor and network analysis schemes is provided in Tables 2 and 3. The actor network analysis does not so much focus on the system of interest to the stakeholder, but rather on the wider policy arena, in which the problem owner has to solve his problem (see Figure 9 below). The relevant social, institutional, and political attributes found will however often lead to the identification of additional tactics, external factors, or criteria and this may lead to extension or modification of all other elements in the system diagram. We suggest the following five steps in the execution of an actor- and network analysis. Departing from the current

Increase electricity generation capacity

Build renewable resourcecapacity

Install wind farms

Install solar systems

Build fossil fuel capacity

Install mCHP units

Install coal fired plant

Install gas turbines

Figure 5 The means-ends diagram of LightOn.

Figure 6

Figure 7 The causal diagram and its relationship with the system diagram.

The means-ends diagram and its relationship to the left side of the system diagram.

1396

Journal of the Operational Research Society Vol. 62, No. 7

Total electricity demand

Fuel price

+ Installed coal-fired systems

spot market purchases

+

Electricity Shortage

+

-

+

+ Total extra fossil + capacity

Installed mCHP systems

+

+ Installed gas fired turbines

+

Net income on sold electricity

+

Total capacity +

Total extra sustainable capacity

+ Installed solar systems

Amount of electricity delivered

+

+ + +

+

total cost (investment plus operational)

Installed wind turbines

+

Figure 8

Table 2

Amount of sustainable capacity

A first system diagram for LightOn.

Part of actor analysis table for LightOn

Actors

Interests

Desired situation/objectives

Existing or expected situation/gap

Causes

Possibilities to influence/courses of solution

Ministry of the Environment

More sustainable energy industry

Less emissions of CO2

Increase in CO2 emissions without adequate policy

Too much use of fossil fuels

Make subsidies available for innovative solutions

Ministry of Economic Affairs

Low energy prices to spur economic growth

More capacity

Capacity shortage causes high prices

Lack of generation capacity

Subsidized investments in power generation

Households

Low energy bill and high security of supply

Pay less for energy and possibly help environment

Higher energy bill

Increase in oil price

Install more energy saving equipment

knowledge about the problem, the actors that might have an interest in the issue are identified. Relevant actors are those who have a stake in the solution of the problem, those who are affected by the solution and those who have a legitimate interest (Enserink, 2000a; de Bruijn et al, 2002; Bryson, 2004). Interviews or reports are a good starting point for the list. Another method is the ‘reputation method’, where actors, starting with the problem owner, are being asked to indicate

who else has the resources that can contribute to or may be affected by the solution of the problem. Second, an outline of the formal chart is developed: the formal tasks, authorities, and relations of actors. Studying the formal legislation, procedures, policy pieces, and so on, provides a complementary indication of the parties that are possibly—or may need to be—involved. Third, the interests, objectives, and problem perceptions of the listed actors are determined. For each

TE van der Lei et al—Using a systems diagram to analyse and structure complex problems

stakeholder the following elements are mapped: interests, desired situation/goals, present or expected situation, causes as seen by the stakeholder, means or possibility to influence the course of events (eg, ‘blocking power’) (see Table 1). Fourth, the interdependencies between the listed actors are explored, based on insight into the resources of the stakeholders. Resources might for example be money, power to implement or block certain decisions, expertize, or gateway to the media. Careful analysis of the causal links between such resources (or means) and criteria may help in identifying and specifying the most important dependencies. Actors who possess resources that are essential for solving the problem owner’s problem are termed ‘critical’ (see Table 2). The determination or willingness to use these resources determines whether such an actor should be considered dedicated or non-dedicated. Fifth, the consequences of these findings with regard to the problem formulation are determined. The lesson is that critical actors are important—their concerns and issues are relevant to the problem owner as their cooperation is needed. Therefore, it is relevant to take their objectives into account in any further analysis, and this may be done by adding additional criteria that show at the right side of the system diagram. The analysis might also have lead to

Figure 9 Policy arena, system diagram, and its relation to the stakeholder network.

Table 3

reconnaissance of additional means in possession of supportive actors that might be willing to invest in the solution of the problem. Those means might be deployed and could be placed as inputs at the left side of the diagram.1 Note that the addition of new elements (criteria and steering factors) will necessitate reconsideration and mostly extension of the causal diagram, and may also lead to the identification of new external factors. Tables 2 and 3 illustrate the possible results of an actor network analysis for LightOn. For reasons of space we limit ourselves to three actors only: the ministry of the environment, economic affairs, and the households. Other examples of actors are: network companies, environmental groups, and producers of different types of energy generation technologies. We assume that the Kyoto protocol forces the national government to reduce the overall carbon dioxide (CO2 ) emissions, and the ministry of the environment is the prime champion of this objective. The ministry could, for example, choose to subsidize sustainable energy generation to reduce the nation’s CO2 emissions. In the assessment of criticality (see Table 2) the households are considered critical but non-dedicated and the two ministries as critical and dedicated, but in different positions when it comes to support for traditional power generation. The households are critical as they can switch to another supplier thus affecting the LightOn’s business position. The ministry of the environment is judged to potentially support LightOn’s when its investment is in line with the ministry’s CO2 reduction policy. Consequently a potentially interesting new means might be government subsidies, represented as a steering factor on the left side of the system diagram. Also, 1 Clearly what is explained here is a rough sketch only. Such a rough sketch misses all nuances, and gives only a snapshot of the situation at a specific time while situations may change rapidly. But the scheme does provide valuable insight in the requirements for and type of process that needs to be organized and gives a glance towards the recommendations.

Table of dedicated and non-dedicated actors for LightOn Dedicated actors

Similar perceptions, interests, and objectives

Non-dedicated actors

Critical actors

Non-critical actors

Critical actors

Non-critical actors

Actors that will probably participate and are potential allies.

Actors that will probably participate and are potential allies.

Indispensable potential allies that are hard to activate.

Actors that do not have to be involved initially.

Ministry of Economic Affairs Different perceptions, interests, and objectives

1397

Potential blockers of (certain) changes. (biting dogs) Ministry of the Environment

Households Potential critics of (certain) changes. (barking dogs)

Potential blockers that will not immediately jump to action. (sleeping dogs)

Actors to whom no attention needs to be given initially.

1398

Journal of the Operational Research Society Vol. 62, No. 7

Figure 10 Visualisation of scenario logic for LightOn.

an additional indicator for the environmental friendliness of the company (CO2 emissions prevented) is added to the right side of the system diagram (see Figure 11).

Question 6: How might the problem change in future? Most policy decisions are about affecting the future, and it is important to consider possible future changes and uncertainties more closely. Will the problem persist, worsen or disappear? Under what conditions? There are two main causes of uncertainty analysis should focus on: (1) changes from within the system, and (2) changes in the context or environment of the system (see Walker et al, 2003), for a more extensive discussion of uncertainty types and sources). A range of methods can be of help to explore the impacts of possible uncertainties on future problem developments. Expert views, trend extrapolation, model-based sensitivity analyses, and a variety of scenario methods may be used (Porter et al, 1991). In our educational setting, we generally focus on the use of contextual scenarios as we feel exogenous sources of uncertainty are generally important and influential but often underestimated or even ignored. Contextual scenarios try to map the effect of the driving forces that have an influence on the system (Schwartz, 1991; Reibnitz, 1998) but largely beyond the span of control of the problem owner. Consequently these scenarios provide images of possible future environments of the system that can influence the results of a policy to a large extent. Context scenarios are generally used to investigate the robustness of the proposed policies and are usually qualitative in nature (Rosenhead, 1989). In our systems approach to problem structuring a small set of alternative possible future contexts for the system of interest is developed, and it is explored how these different

contexts would affect the problem situation (Thissen, 1998; Enserink, 2000b, 2004). As a result, sometimes new exogenous factors need to be taken into account. Figure 10 presents a (simple) scenario logic or scenario skeleton for the LightOn example. The three axes span a scenario space wherein an infinite number of scenarios can be positioned. For LightOn we developed a limited set of contextual scenarios that were further detailed, inspired by Schwartz’ approach to scenario composition (Schwartz, 1992). The various steps presented above lead to either specification of elements of the problem analysis, or modification, for example by adding criteria, exogenous factors, or means. In principle, each of these additions may affect other elements of the problem analysis. Therefore after each step an iteration is made to check and compensate for inconsistencies. For example, a newly added tactic may affect an actor who had not been considered thus far. Adding that actor may lead to reconsideration of the set of criteria, and this in turn may require revision of the causal diagram. Therefore, after performing the preceding steps, a consistency check is advised to verify whether all relevant actors and factors have been included. The result is portrayed as a ‘final’ system diagram, and is accompanied by a verbal reformulation of the problem in terms of both needs and choice dilemma’s. Figure 11 illustrates the diagram for our simplified LightOn example after several iterations and consequential revisions and updates. During the analysis process, possible extension of the set of problem owners needs to be considered. We started from the perspective of the client as the single problem owner. As illustrated in the LightOn case the actor analysis may show that some of the actors in the policy arena possess means that are relevant to solving the problem, and that these actors may be willing to apply those means in accord with the problem owner. In such cases, such actors may be considered as coproblem owners. Then, the analysis and the corresponding diagrams should be adapted in accordance. Consequently the factors under control of the added problem owners should be added as steering factors at the left side of the system diagram, and their objectives should be operationalized into criteria to be added as necessary. In practical real-world situations, choices like this should be discussed with and agreed with the problem owner(s).

Question 7: What further steps, including possible research, are recommended? The final step in the policy issue paper is to identify a plan of action to be recommended to the problem owner. Two different types of actions may be distinguished: so-called process actions, for example to start talking to or negotiating with critical actors, and research actions, targeted at resolving specific crucial knowledge gaps. To identify relevant knowledge gaps, a ‘walk through’ of the system diagram is done in order to assess what the analyst

TE van der Lei et al—Using a systems diagram to analyse and structure complex problems

+

Installed coal fired systems

Electricity Shortage

Installed mCHP systems

+

Total extra fossil capacity

+ Installed gas fired turbines

Installed solar systems

Installed wind turbines

Technological progress renewables

Total electricity demand

Fuel prices

1399

+

+

Spot market purchases

+

+

Amount of electricity delivered

+ +

+

+ Total capacity

Total extra sustainable capacity

+

+

-

? + Total cost (investment plus operational)

+

CO2 emissions prevented

-

?

+ +

+

Net income on sold electricity

Amount of subsidy

+

Subsidy

Amount of sustainable capacity

Figure 11 The final system diagram for LightOn.

knows and does not know about the system. This is done by following the impact of the policies through the factors in the system on the criteria. Attention should be drawn to those relationships between factors that are insufficiently known. Typically there are unknowns about the causalities within the system, the impacts of policies on the criteria, the influence of other stakeholders, and how the problems may change over time. The knowledge gaps that have been identified are then translated into one or more main research questions that can be split into a number of detailed sub-questions. Preferably this is done in consultation with the problem owner and the critical actors. For each sub-question a research proposal is formulated specifying, among other things, the method(s) to be used (Van der Lei and Slinger, 2006). A timetable and a budget estimate can also be incorporated for the potential commissioner of the studies. Another important element is a communication plan stating the form and level of cooperation and involvement of problem owner and stakeholders and the way reporting will be organised. The analysis performed for LightOn has shown that investments in sustainable capacity seem to be important for the image and long-run success of the company. Until now insight

into the specific costs per capacity type are lacking. As a research question we pose: What are the investment costs per type of sustainable capacity for LightOn? As the choice of capacity is an investment problem a cost-benefit analysis is recommended. In addition, given the significant uncertainties regarding technology, fossil fuel prices, etc, we recommend to perform a more thorough and complete uncertainty analysis.

Discussion There is no unique way to analyse a complex policy problem. We have taken system analytic thinking as a starting point, emphasising the substantive aspects of a problem situation, and the logic and internal consistency of the conceptual model(s) developed. Answering the seven questions formulated in Table 1 generates the insights in the problem that need to be reported in the policy issue paper. Attractive about this stepwise approach is the coherence of the models used and produced. By using simple analytical modelling techniques to construct the systems model its internal consistency is guaranteed. By combining them with actor analysis and uncertainty analysis an integrated model is build up, wherein content and process aspects are connected. This iterative

1400

Journal of the Operational Research Society Vol. 62, No. 7

process of problem formulation and reformulation guided by the questions leads to a balanced problem formulation and delineation. Political scientists, however, would generally rather focus on the broader policy arena first (see Figure 9). Such an analysis of actors, their perceptions, intentions and behaviour, may lead to quite different framings of problem situations, for example, the absence of trust in a policy network may be seen as a prime impediment of progress, or the problem is framed as a struggle for power between different parties. Such different approaches may lead to complementary and relevant insights, and the nature of further action required to reach a solution (if possible at all) will generally depend on the nature of the situation, who the problem owner is, etc. For further exploration of such different approaches, the reader is referred to the literature (Bobrow and Dryzek, 1987; Dunn, 1994; Mayer et al, 2004). Limiting ourselves to the system analytic approach outlined above, we identify some of the recurring difficulties encountered in the period of more than 10 years in which we have been working with and teaching the approach to several generations of students. Table 4 below lists some of the key issues and choices that time and again appear to be difficult both for us teachers (when developing examples) and students alike. In the table, we also have added a number of tips and suggestions for dealing with these issues. Note that some of the

Table 4

issues listed relate to the way results from the analysis are documented and reported in a policy issue paper. We have developed this approach when faced with the need to teach our students a systematic, analytic approach to tackling complex policy problems, and found little ready-to-use material in existing texts. Inspired by some of our own realworld experiences, we set out to systematically combine some of the parts of the puzzle we found in literature, and adapted them to be applicable at the undergraduate level. Although no systematic empirical testing in real-world contexts has been done yet, we believe the approach to be of value to practicing analysts when faced with complex, unstructured situations. We see our students often successfully apply the approach in their MSc thesis work which is mostly done in a real-world context, and we regularly obtained positive feedback of alumni about how this part of their education has benefited them in structuring their work as practicing analysts.

Concluding remarks We have outlined how a conscious and step-wise combination of relative simple and straightforward analytical methods can be used to frame a complex policy problem in the form of a systems diagram. Seven questions accompanied by relatively straightforward analytical methods and models constitute the

Typical problems and ways to tackle these

Steps or questions

Problems and common mistakes

Tips and tricks

Problem level

The problem is framed uniquely as a choice between alternatives It is difficult to choose an appropriate scope for the primary objective(s). Sometimes, there is a tendency to choose for very abstract high-level objectives like ‘increase of welfare of all inhabitants’ or ‘economic growth’

Emphasise that a problem is a gap between desire or wish and (expected) reality Start thinking backwards from the chosen objective to identify means; if the primary objective is too broad, the spectrum of possible means and tactics becomes overwhelming

Sometimes, too narrow a scope is chosen (jumping to solutions)

Ask the ‘next general question’: Why does the problem owner want to achieve the initial objective? Consider taking the next higher level objective as starting point instead as it may allow for a broader set of options

Choice and analysis of objectives

The elaboration of objectives into an objectives hierarchy and operational criteria is mixed up with a means-ends analysis exploring ways or means to achieve the objectives

Emphasise the difference between a definition relation (what) and a causal (or intentional) relation (how). Use appealing examples. Use nouns for objectives and verbs for means

Identifying the means of the problem owner?

It is tempting to extend the explorations of means to the elaboration of implementation measures

Emphasise the need to focus on steering factors first, that is, the key things or factors a problem owner can influence to change the situation. Elaboration of implementation measures can follow later, after analysis has helped identify the most promising tactics and strategies (see also Walker, 1988) Limit attention to strategies that have a direct effect on factors in the system; focus on the main types of actions only, elaboration and fine tuning may follow later

Too many tactics and strategies

TE van der Lei et al—Using a systems diagram to analyse and structure complex problems

Table 4 Steps or questions

1401

Continued

Problems and common mistakes

Tips and tricks

Sometimes it is unclear where the factors belong. Are Determining system boundaries they part of or outside the system? and elaborating the system diagram The question what the problem owner can influence and what is under the influence of other actors is sometimes hard to answer.

A large number of factors inside the system diagram with a lot of causal relationships between them

Emphasise as a basic rule that factors only belong in the system if they can be affected by means or tactics under the control of the problem owner(s) It’s a matter of judgment and choice; best is to make a choice and be explicit about it. If other actors may be influenced by the problem owner, an option is to include them as part of the system and consider the action to influence them as a tactic Limit the number of factors to a manageable size (about 15) to prevent ‘spaghetti diagrams’ and to keep oversight and to support the self-explanatory character of the diagram. An alternative is to identify subsystems rather than factors

What further research is recommended?

Research is suggested on resolving the exogenous This may seem logical as exogenous factors often uncertainties as these appear to be of crucial importance represent the most influential uncertainties; however, often more research will not easily resolve these uncertainties (eg regarding climate change, energy prices, etc), so the advice should be to accept these uncertainties and focus research on identifying those options that best deal with the uncertain environment

General

Different problem definitions ‘float’ through the paper. The argument ends with the same problem description as it started, with no clear value added from the analysis Suddenly new problems and new variables are introduced that do not logically follow from the analysis

backbone of a sound problem analysis. The insights derived are the input for the proposed policy issue paper we expect our students to write. The approach focuses on the content dimension of policy problems, and relates this to the multi-actor environment within which most policy issues are debated. By iteration the approach leads to an internally consistent systems model that represents the problem definition and delineation. Scrutinising the factors and the character of their relations leads to the identification of knowledge gaps requiring further scientific, analytical or political research. The interrelation between the various analytical steps and the subsequent iterations force the policy analyst to continuously re-evaluate the problem definition and problem delineation. Guarding consistency—checking whether the criteria on the right side of the systems diagram are indeed the same as the operationalised lowest level objectives in the objectives hierarchy, whether the steering factors match the means, to what extent causal mechanisms relating the means and the criteria are made explicit, and to what extent the actors related to the various factors and tactics have been identified—helps the analysts to construct an internally consistent and convincing formulation of the most relevant problem situation elements, and their relations. This then provides a basis for discussion with the problem owner(s) and for planning further steps.

Check for consistency in the storyline; make one sheet describing the main line of argumentation Either these issues should be mentioned as insights from earlier analytical steps or as factors showing up in previous models and diagrams or they should be left out

The systems perspective illustrated in our integrated problem structuring approach is expected to lead to clear and sound insight in the character and complexity of problems; to better problem descriptions and consequently better proposals for follow-up.

References Ackoff RL (1978). The Art of Problem Solving. Wiley: New York. Axelrod R (1976). Structure of Decision: The Cognitive Maps of Political Elites. Princeton University Press: Princeton, NJ. Bobrow DB and Dryzek JS (1987). Policy Analysis by Design. Pittsburgh University Press: Pittsburgh. Bryson JM (2004). What to do when stakeholders matter. Stakeholder identification and analysis techniques. Public Mngt Rev 6(1): 21–53. Bryson JM, Ackerman F, Eden C and Finn CB (2004). Visible Thinking. Unlocking Causal Mapping for Practical Business Results. John Wiley & Sons: Chichester. Checkland PB (1985). Formulating problems for systems analysis. In: Miser HJ and Quade ES (eds). Handbook of Systems Analysis, Overview of Uses, Procedures, Applications, and Practice. North-Holland: New York, pp 151–170. Checkland P and Poulter J (2006). Learning for Action: A Short Definitive Account of Soft Systems Methodology and Its Use, for Practitioners, Teachers and Students. Wiley: Chichester.

1402

Journal of the Operational Research Society Vol. 62, No. 7

de Bruijn H and Porter AL (2004). The education of a technology policy analyst to process management. Technol Anal Strateg Mngt 16(2): 261–274. de Bruijn H, ten Heuvelhof EF and in’t Veld R (2002). Process Management: Why Project Management Fails in Complex Decision Making Processes. Kluwer Academic Publishers: Dordrecht, the Netherlands. Dunn WN (1994). Public Policy Analysis: An Introduction. Prentice-Hall: Englewood Cliffs, NJ. Eden C (1988). Cognitive mapping. Eur J Opl Res 36: 1–13. Eden C (2004). Analyzing cognitive maps to help structure issues or problems. Eur J Opl Res 159: 673–686. Eden C and Ackermann F (1998). Making Strategy: The Journey of Strategic Management. Sage: London. Eden C and Ackermann F (2006). Where next for problem structuring methods. J Opl Res Soc 57: 766–768. Enserink B (2000a). A quick scan for infrastructure planning: Screening alternatives through interactive stakeholder analysis. Impact Asses and Proj Appraisal 18(1): 15–22. Enserink B (2000b). Building Scenarios for the University. Int T Opl Res 7(6): 569–584. Enserink B (2004). Thinking the unthinkable—The end of the Dutch river dike system? Exploring a new safety concept for the river management. J Risk Res 7(7–8): 745–758. Goodell J (2001). How coal got its glow back. New York Times, http://www.blackmesais.org/How Coal Got.html, 22 July, accessed July 2007. Harrison J (2004). Micro combined heat & power (CHP) for housing. SET 2004—3rd International Conference on Sustainable Energy Technologies. Nottingham. UK, 28–30 June, http://www.microchp.dk/Jeremy%20Harrison%20micro%20CHP% 20SET%202004.pdf, accessed July 2007. Hoogerwerf A (red.) (1988). Het ontwerpen van beleid. Een handleiding voor de praktijk en resultaten van onderzoek, Samsom H.D. Tjeenk Willink, Alphen aan den Rijn. Keeney RL (1988). Structuring objectives for problems of public interest. Opns Res 36: 396–405. Keeney R (1992). Value-focused Thinking. A Path to Creative Decision Making. Harvard University Press: Cambridge, MA. Keeney RL and Gregory RS (2005). Selecting attributes to measure the achievement of objectives. Oper Res 53(1): 1–11. Koppenjan J and Klijn EH (2004). Managing Uncertainties in Networks. Routledge: London, New York. Mayer IS, van Daalen C and Bots PWG (2004). Perspectives on policy analyses: A framework for understanding and design. Int J Technology, Policy Mngt 4: 169–191. Miser HJ and Quade ES (1988). Handbook of Systems Analysis, Craft Issues and Procedural Choices. John Wiley & Sons: Chichester, New York, Brisbane, Toronto, Singapore. Porter AL, Roper T, Mason T, Rossini F and Banks J (1991). Forecasting and Management of Technology. Wiley & Sons: New York. Quade ES (1980). Pitfalls in Formulation and Modelling. In: Majone G and Quade ES (eds). Pitfalls of Analysis. John Wiley & Sons: Chichester. Quade ES (1982). Analysis for Public Decisions, 2nd edn. NorthHolland: New York. Quade ES (1989). Preparing an issue-paper. In: Analysis for Public Decisions. North Holland: New York, pp 73–78. Rittel WJ and Webber MM (1973). Dilemmas in general theory of planning. Policy Sci 4: 155–169.

Rosenhead J (1989). Robustness analysis: Keeping your options open. In: Rosenhead J (ed). Rational Analysis for a Problematic World. John Wiley & Sons Ltd: Chichester, West Sussex, England. Rosenhead J and Mingers J (eds) (2001). Rational Analysis in a Problematic World Revisited. Wiley: Chichester. Sage AP (1992). Systems Engineering. IEEE Wiley: New York. Sage AP and Olson SR (2001). Modeling and simulation in systems engineering. Simulation 76(2): 90–91. Schwartz P (1991). The Art of the Long View: Planning for the Future in an Uncertain World. Doubleday: New York. Schwartz P (1992). Composing a plot for your scenario. Planning Review 20(3): 4–9. Shaw D, Franco A and Westcombe M (2006). Problem structuring methods: New directions in a problematic world. J Oper Res Soc 57: 757–758. Simon HA (1952). On the definition of the causal relation. J Philos 49: 517–528. Smaardijk EJ, Fliert EM, Ledeboer A and Jak MJG (2005). Disperse energie—Liberale Markt, ALTRAN in opdracht van Provincie Zeeland en Delta N.V. Sterman JD (2000). Business Dynamics. Systems Thinking and Modeling for a Complex World. McGraw-Hill Higher Education: Boston. Thissen WAH (1998). A scenario approach for identification of research topics. In: Weijnen MPC and Heuvelhof EP ten (eds). The Infrastructure Playing Field in 2030. Proceedings of the First Annual Symposium Delft Interfaculty Research Center Design and Management of Infrastructures, November, Delft University Press; Noordwiik, pp 5–10. Thissen WAH (2000a). Systems engineering education for public policy. Int J Technol Mngt 19(3–5): 408–419. Thissen WAH (2000b). Issue formulation in a multi-actor context: A five-step approach. In: Kawamura K (ed). Proceedings 2000 IEEE International Conference on Systems, Man, and Cybernetics, Vol. 1, Nashville, TN, USA, pp 301–306. Van der Lei T and Slinger JH (2006). The system diagram as a diagnostic aid for quantitative method choice. In: Antunes CH and Gomes A (eds). CDRom Proceedings, ORMMES 2006 Operational Research Models and Methods in the Energy Sector, 6–8 September 2006, Institute of Systems Engineering and Computers of Coimbra and Association of European Operational Research Societies; Coimbra, Portugal. von Reibnitz U (1998). Scenario Techniques. McGraw-Hill: New York. van Walle E, director-general of the Belgian Nuclear Research Centre SCK-CEN (2006). Speech on the Strategic Forum on Energy Supply in the 21st century, http://www.leonardo-energy .org/drupal/node/1530, Brussels, 18 December 2006, accessed March 2010. Walker W (1988). Generating and screening alternatives. In: Miser HJ and Quade ES (eds). Handbook of Systems Analysis. John Wiley & Sons: Chichester, New York, pp 217–234. Walker W (2000). Uncertainty: The Challenge for Policy Analysis in the 21st Century. TU Delft, Faculteit Techniek, Bestuur en Management: Delft. Walker WE, Harremo¨es P, Rotmans J, van der Sluijs JP, van Asselt MBA, Janssen P and Krayer von Krauss MP (2003). Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integr Assess 4(1): 5–17.

Received November 2008; accepted January 2010 after two revisions