Journal of the Operational Research Society (1998) 49, 445±457
#1998 Operational Research Society Ltd. All rights reserved. 0160-5682/98 $12.00 http://www.stockton-press.co.uk/jor
A survey of knowledge-based systems research in decision sciences (1980 ±1995) R Santhanam and J Elam Florida International University, USA Research in knowledge-based systems (KBS) has become an important area of inquiry within decision sciences. In this paper, we present the results of an extensive survey of research papers published on this topic. We determined frequency counts of papers and we also performed a content analysis of the papers we surveyed. The results indicate that there are a large number of studies informing us of the design and development issues relating to KBS. However, there seems to be less research examining issues relating to the management and impact of KBS on individuals and organisations. We summarise our key ®ndings and identify avenues for future research. Keywords: knowledge-based systems; decision support systems; arti®cial intelligence; literature survey; decision sciences
Introduction In the last twenty years, the area of Arti®cial Intelligence (AI) has generated great interest in science, academia and business organisations. The power and importance of knowledge in supporting business problem solving have been recognised and this has given rise to the development of knowledge-based systems. For the purpose of this study, we will de®ne a knowledge-based system (KBS) as one, which utilises AI methods and stored knowledge of a speci®c problem or technique to provide decision support. KBS are considered to be the most popular and successful form of AI systems and have `attained a permanent and secure role' in business organisations today.1 Consequently, researchers have investigated issues relating to the design, development and management of these systems. The significance and importance of this topic are evidenced by the numerous meetings, journal articles and special issues of journals dedicated to reporting research in this area. Leading business journals such as, Decision Sciences and European Journal of Operational Research have devoted special issues of their journals to publishing research pertaining to KBS.2±4 These are indicative of the growing impact of KBS in in¯uencing managerial decision making in organisations and research efforts in the ®eld of decision sciences. The ®eld of Decision Sciences encompasses tools, methods, and systems that aim to improve problem solving and decision making in business organisations. As a discipline, it utilises the analytical models and techniques developed by Management Science/Operations Research (MS/OR) Correspondence: Dr R Santhanam, Department of Decision Sciences and Information Sciences, Florida International University, University Park, Miami, FL 33199, USA. E-mail: Santhana@servax.®u.edu
researchers and incorporates them into decision-aiding systems using the information technologies provided by Information Systems (IS) researchers. This intersection between analytical techniques and information technologies has resulted in powerful decision aiding tools and systems such as Decision Support Systems (DSS) and Expert Systems (ES).2,3 It is not surprising, given the decision making focus embodied in knowledge-based systems that research in KBS has expanded into the decision sciences ®eld from its original roots in computer science. The purpose of this paper is to present the results of a survey that examines KBS research published in decision science journals. While journals in other disciplines also publish articles on KBS, we limit our analysis to decision science journals because our primary objective is to provide information to researchers and practitioners who are interested in this area. Secondly, our research methodology entailed a detailed content analysis of articles that we surveyed. We had to draw a boundary to make this analysis meaningful as well as manageable. Since this survey examines research in KBS from a decision science's perspective, only journals in MS/OR, IS, and DSS are used as the basis for the survey reported in this paper. As a new ®eld of study grows and achieves maturity, a survey of the ®eld serves several important purposes. Firstly and foremost, it documents the breadth and signi®cance the area has attained. Secondly, it classi®es the topics in the ®eld that have been studied and provides some trend analysis. Thirdly, it highlights important issues that have not been addressed by researchers and thus provides a platform for emerging research directions. From the MS/OR and IS literature, 430 articles relating to KBS published between 1980 and 1995 were examined and classi®ed according to a prede®ned classi®cation
446 Journal of the Operational Research Society Vol. 49, No. 5
scheme. Articles related to a particular topic area were analysed and a summary of research ®ndings by topic area was created. The rest of the paper is organised as follows. Firstly, the methodology used to identify and classify articles is described. Results from the classi®cation scheme and from the content analysis of articles are presented next. Implications of these results for current and future research in KBS are then discussed. Methodology Scope of the survey The purpose of this survey was to examine the MS/OR, DSS, and IS literature from 1970 ±1995 that focused on the design, development, use, and impact of speci®c KBS as well as the incorporation of KBS components into traditional individual and group decision support systems. We did not ®nd any published article in the years 1970 ±1980 in the target journals. While there was a lot of interest in KBS during these years, research studies were probably published in computer science oriented journals. Secondly, while research was being presented in conferences it had probably not reached the target journals. Hence, 1980 became the actual starting point for our analysis. The process of selecting journals from the MS/OR, DSS, and IS literature to be used as the basis for this survey began with the list of 22 journals used in a review of the DSS literature.5 We added two journals to this list, namely, European Journal of Operational Research (EJOR) and Journal of the Operational Research Society (JORS) that publish papers on operations research and KBS. We also added Information Systems Research (ISR) that started publication in 1990. From this list of 25 journals, we eliminated seven journals such as Journal of Accounting Review, Journal of Accounting Research, Computer, etc., that did not directly relate to MS/OR or information systems. The ®nal list contained 18 refereed journals representing all major publications in the MS/OR and information systems ®elds. The abstract and the ®rst page of every article published in the journals selected for this study was examined by the authors and included if it pertained to original research on KBS. Reports, opinions, status report, forewords to journal papers, and editorial comments were not included in the survey. Articles that utilised AI technology but did not pertain to KBS applications in business decision-making (such as those relating to robotics, computer aidedengineering, etc.) were not included. Classi®cation scheme A classi®cation scheme composed of two main categories was used to describe every article included in the survey. Each article was assigned a value for each of the two
categories. The categories were article type and topic. A description of each category is given below. Article type: Articles included in this survey can be divided into three types: articles whose results follow from the application of a traditional empirical research methodology; articles that are tutorial in nature; and articles that are conceptual in nature. Within the ®rst type of article, three speci®c research methodologies were identi®ed. Within the third type of article, two different types of conceptual articles were identi®ed. These resulted in seven speci®c types of articles as described below. (1) Case studyÐArticles that provided insights into the process of design, development, implementation, and/or use of a KBS in actual organisational settings from interviews with key individuals within these organisations or from actual experiences of authors in developing and implementing a KBS. (2) Survey ÐArticles that analysed data collected from questionnaires, or analysed the literature. (3) ExperimentÐArticles that were based on the results of laboratory or ®eld experiments. (4) Tutorial ÐArticles that provided explanations and descriptions of KBS concepts that are known to researchers and practitioners in this area. (5) Proposal and developmentÐArticles that provided speci®c design proposals for a KBS or for tools for developing a KBS. Many of these articles described the prototype development and testing of these design proposals. (6) IdeasÐArticles that presented general ideas for KBS design, development, and implementation or provided a framework for KBS design and research. Topics: This category identi®ed the major research topic addressed in an article. The articles included in the survey predominantly focused on one of the following topic areas: (1) Design and developmentÐArticles that addressed issues related to the design and development of KBS. These issues include such things as providing guidelines for identifying good KBS applications, methods for knowledge acquisition, assessment of particular AI tools, system architectures, and speci®c design features. For example, an article that described the design of a user interface for a KBS would be assigned to this topic area. (2) ValidationÐArticles that discuss methods for determining the performance level of a KBS and/or the accuracy and reliability of the KBS. For example, an article that describes speci®c validation techniques would be assigned to this topic area. (3) ManagementÐArticles that focus on issues relating to the management of the KBS development and implementation process. For example, an article that
R Santhanam and J ElamÐA survey of knowledge-based systems research in decision sciences (1980±1995)
describes KBS project management techniques or techniques used in the actual deployment of a KBS would be assigned to this topic area. (4) Use and impactÐArticles that discuss issues relating to the impact of KBS on individual and organisational decision making, ®nancial impacts of a KBS on an organisation, and/or the general use of KBS in business. For example, an article that examined how a KBS could be used to support production planning activities would be assigned to this topic area. Validation of classi®cation scheme The classi®cation scheme used in this study was ®rst pretested on a random set of articles. Based on this pretest, de®nitions of each category and rules for inclusion were slightly modi®ed and made more precise. Each paper was classi®ed separately by the two authors of this paper. The authors then met and compared the classi®cation. Overall, there was a high level of agreement between the authors. Discrepancies that existed were resolved through mutual discussion. Each article was classi®ed into only one category that was most representative of its content. Content analysis In addition to classifying each article for each of the three categories described above, a brief summary for each article that described its main content and contribution
447
was prepared. These were then summarised to highlight common themes and results. Results from classi®cation of articles A total of 430 articles were included in this survey. The results of the classi®cation are shown in Tables 1±3. There were no articles found in the Academy of Management Journal and the Administrative Science Quarterly and hence they are not shown in the tables. As seen in Table 1, there were only 26 articles published during the years 1980±1985. Between the years 1986 and 1990, there was a sudden increase and 205 KBS articles were published during this period. This large increase in the number of articles published indicates an accelerating research interest in KBS, perhaps in¯uenced by large, sponsored projects such as the ®fth generation project in Japan in 1981 and the Alvey Initiative in United Kingdom in 1982. It can also be attributed to the start of new journals such as Decision Support Systems that publish many papers in this area. Table 1 shows that Decision Support Systems has published the most articles, followed by European Journal of Operational Research, Journal of the Operational Research Society and Communications of the ACM. In terms of Article Type, Table 2 indicates that the vast majority of the articles on KBS are of a conceptual nature. Sixty two percent of the papers propose a speci®c KBS with differing degrees of actual prototype development. Eighteen percent of the papers present new ideas or general frameworks for the development of a KBS. Tutorials
Table 1 Frequency by Journal and Year 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 Total Communications of the ACM Data Base Decision Sciences Decision Support Systems European Journal of Operational Research Harvard Business Review Information & Management Information Systems Research Interfaces Journal of Management Information Systems Journal of the Operational Research Society MIS Quarterly Management Science Omega Operations Research Sloan Management Review Total a
Special issues relating to KBS.
1a 1 1 1
6a 1
1 2
8
15a 1
1 3
1
4
20
4a
2 8a 1
2 7 21a
4 2
3 5
3 1
3
3
2
3 1
3
27 22
2
1
1
3
1
42
1
2 1
1 1 1 1
2 2 1
3 3
43
34
31
16 9 10 3 3 430
1 2
1 2
3
2 1
7
2 1a
4 6
8 2a
2 1
4 3
3
5
5
2
4
15a
1 1 1 1
2
4
2 1
4a
8a 4
1 30
1 1 42
51
1 1 1 12 5
1 1 5 10a 2
7a 8 4
2
5 2
11a 4 2 3 1 14a 11a 8 1 3a
1 2 1 24a 4
1
3
2 33
2 49
1 44
47
42 14 34 119 47 2 31 9
448 Journal of the Operational Research Society Vol. 49, No. 5
Table 2 Article classification by year Article type Years 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 Totals Percent
CS
SRY
EX
1 1 3 2 4 4 5 2 2 3 1 27 6
1 3 1 3 2 2 2 2 17 4
1 2 4 1 4 2 4 18 4
TT
1 6 4 4 3 2 1 1 1 2 25 6
Topics P/D
1 1 11 19 23 30 19 29 32 30 19 16 35 266 62
ID
Total
D&D
1
1 0 0 1 4 20 30 42 51 33 49 44 43 34 31 47 430 100
1
1 3 5 11 11 7 8 4 8 7 6 6 77 18
Article type CSÐCase Study SRYÐSurvey EXÐExperiment TTÐTutorial P/DÐProposal/Development IDÐIdea
account for another 6%. Given the newness of the KBS ®eld, it is not surprising that there have been a number of tutorials published. There are a relatively small number of articles (10%) that use laboratory experimentation or ®eld survey research methodology in the research reported. About 6% of the articles surveyed use a case study as a research method. Given that the majority of articles were of the Proposal/ Development type, it is not surprising that the topic most often addressed by KBS research is design and development. As shown in Table 2, 59% of the articles discussed and provided ideas relating to technical aspects of building KBS. About 34% of the papers focused on the use and impact of KBS. Other topic areas were only minimally represented (5% related to Management and 2% related to validation). Table 3 shows the relationship between journal and article type and topic. As can be seen from this table, research on proposal/development has been published in all of the journals except for the Harvard Business Review and the Sloan Management Review. Only tutorials have been published in the Harvard Business Review and only a case
VAL
MGMT
1 15 21 23 28 10 26 31 32 23 15 28 254 59
1 2 1 1 2 1 1 1 10 2
2 2 3 2 3 3 3 1 1 3 23 5
U&I
Total
4 3 7 15 19 20 19 9 9 9 14 15 143 34
1 0 0 1 4 20 30 42 51 33 49 44 43 34 31 47 430 100
Topics D&DÐDesign and Development VALÐValidation MGMTÐManagement U&IÐUse and Impact
study and idea paper in Sloan Management Review. The Journal of the Operational Research Society is the only journal that has published all article types although journals such as Communications of the ACM and Information and Management have published all article types except experiments. Results from content analysis The information obtained from the classi®cation of articles provides a useful overview of the nature and content of KBS research. For each of the research topic areas, we summarised the major conclusions obtained from research on that topic and current issues that are being examined. We then identify some issues that can be investigated in future research. Design and development (D&D) The majority of surveyed papers were published on this topic. These papers address a wide variety of issues
R Santhanam and J ElamÐA survey of knowledge-based systems research in decision sciences (1980±1995)
449
Table 3 Article classification by journal Article type Journal Communications of the ACM Data Base Decision Sciences Decision Support Systems European Journal of Operational Research Harvard Business Review Information & Management Information Systems research Interfaces Journal of Management Information Systems Journal of the Operational Research Society MIS Quarterly Management Science Omega Operations Research Sloan Management Review Totals
CS
SRY
1 2
4
2
2 2
5
7
3 1
1
3
1
4 1 3 2 27
EX
3
TT
P/D
ID
Total
D&D
6 1
26 7 24 101 29
5 4 7 16 10
42 14 34 119 47
30 6 23 93 22
3 3 10 4
2 31 9 27 22
2 14 8 7 12
2
4
4 2 2
2 1
4
7 1
13 4 6 12
2
3
28
5
42
21
4 6 4 2
4 1 3 1 1 77
16 9 10 3 3 430
6 7 2 1
4 1
17
Topics
18
25
266
Article type CSÐCase Study SRYÐSurvey EXÐExperiment TTÐTutorial P/DÐProposal/Development IDÐIdea
including, how to identify tasks that can be automated with an ES, methods to acquire knowledge, the role and use of logic programming, the role of ES shells, general design principles to be used in KBS development, and how to design a good interface. The feasibility of some of the proposed design principle was also demonstrated by the development of a prototype system. Research papers on systems design for a speci®c domain/application are discussed under the Use and Impact topic. The development of a KBS starts with choosing a problem domain that can be automated with a system. Most of the published systems deal with generic tasks of problem diagnosis, trouble shooting, scheduling, design and con®guration of systems. In the business domain, Sviokla6 and Jain and Chaturvedi7 provide a broad set of criteria that can be applied to see if a given task can be automated with a KBS. They state that a task can be automated only if it is suf®ciently well de®ned and can be broken down into subproblems, conventional programming techniques do not offer a suitable solution, the task has well-
VAL
254
4
MGMT
U&I
Total
2 2
10 6 7 26 23
42 14 34 119 47
6
11 1 19 9
31 9 27 22
2
15
42
6
4 2 7 2 1 143
16 9 10 3 3 430
2
1 1
1 10
2 23
Topics D&DÐDesign & Development VALÐValidation MGMTÐManagement U&IÐUse and Impact
known experts who may not be available in the future, and the heuristics needed to complete the task and corresponding results are very clear. After deciding on the appropriate problem, the next step is the process of acquiring the problem knowledge from domain experts. In many business applications, knowledge may be available within the organisation among a few organisational members. Using two organisational settings, Stein8 shows how to use a network analysis to identify sources of documented and undocumented knowledge within the organisation. This can be achieved by using a specially prepared questionnaire that seeks out people's knowledge on the domain and thus identi®es organisational members who have a lot of expertise. Interviewing is an easy-to-use technique that is most often utilised to acquire knowledge from experts. Interviews take a long time and researchers suggest that knowledge engineers use a structured interviewing technique where they develop a model of the domain before they start to acquire knowledge.9 Seagle and Duchesi10 suggested the use of a decision table comple-
450 Journal of the Operational Research Society Vol. 49, No. 5
mented with a decision analyser as a tool to reduce the tedium of the interview process. Some researchers feel that traditional requirements analysis used in information systems analysis shares many characteristics with knowledge acquisition methods because they both try to overcome communication barriers.11,12 Therefore, knowledge engineers could learn information gathering techniques from systems analysts and vice-versa. Given that the interviewing process is a time-consuming task, automatic rule-induction methods using ID3 algorithm, neural networks, genetic algorithms, concept induction methods have been widely tested.13±16 In these methods, the induction system uses sample data sets where the solution or expert's judgments are known and it generates domain knowledge rules that can be incorporated in the system. These methods seem to be fairly successful in generating knowledge in structured domains such as predicting bankruptcy failures17 and developing loan granting rules.18 However, researchers caution that some of these methods may appear to be very successful on sample cases but not as ef®cient in generating knowledge in real test cases.19 Therefore, more testing is required about the realworld utility of these methods and whether they can perform better than classical methods such as, discriminant analysis and non-parametric methods. Once knowledge is acquired, the development of the system can take place. While traditional programming can be used, it is recommended that symbolic programming based on LISP or Prolog be used. Logic programming languages and logic modeling approaches are two of the most researched topics among the articles we surveyed.20 Instead of programming the system, the use of commercial software called shells are recommended for developing the system. There is a host of such shells available, making it dif®cult for a manager to sort out among the vendors' claims and decide on the appropriate shell to purchase. Also, a bad choice could jeopardise the success of the ES. Therefore, Stylianou et al21 provided a detailed framework that can be applied to commercially available shells to evaluate their suitability for the given task. They conduct a ®eld survey and ®nd that backward chaining, explanation facility, rapid prototyping, and embedding capability are among the most valued features in an ES shell. From the above, it is obvious that there are several developmental approaches that can be used to develop an ES. When in doubt as to what is the most appropriate method to use, Nelson and Balachandra22 suggested the use of an analytical hierarchy process. They show how this method will allow a project manager to systematically examine all the competing development approaches and choose the most appropriate method for the given problem. Researchers caution that not only is the knowledge-base an important component of a KBS, but the design of the interface and the type of explanations provided is also a critical issue affecting end-user acceptance and usage of the
system.23 Empirical results by Lamberti and Wallace24 indicated that the type of presentation format in an ES has to be matched to the skill level of the users such that it easily maps with their conceptual representation of the problem. It was found that experienced users performed better with declaratively formatted presentation in high uncertain tasks while less-experienced users performed better with declarative presentation in less uncertain tasks. Thus, the complexity of explanations from the KBS must be geared to match the skill level of the user. Pei and Reneau25 found similar results that con®rm the need to match users' domain knowledge and their problem representation with the problem solving strategy of the KBS. The explanation facilities/ interface mechanisms can perhaps be made independent of the knowledge base representation (like data independence) and made to match the user's problem solving strategies.24 Other design and development issues Due to the similarity between a DSS and KBS, a few authors suggest that some of the design principles recommended for building a DSS such as, iterative development, adaptive design, and use of representations, can also be used to develop a KBS.26±28 Much research has been conducted to make the model component more intelligent and assist the user to formulate and use mathematical models.29 Addressing issues relating to the design, the interface, the explanation facilities, and the components of such systems has become a distinct stream of research called model management research. A survey of research on this topic is provided in Bharadwaj et al.30 We found fewer papers that investigated the role of the database component in a KBS. Methods to integrate knowledge bases with existing organisational databases,31 and the use of database management tools to retrieve facts from a KBS, are some of the ideas proposed32 to integrate knowledge-base and database technology. KBS and operations research There are a large number of research articles that discuss the advantages of combining AI methods with other quantitative techniques. Speci®cally, the combination of heuristic search methods of AI can be easily combined with the algorithmic approaches of MS/OR and applied for knowledge-based business applications.33±35 The design and development of prototype systems for planning railroad logistics,36 for performing computations in continuous time systems37 and for new product development,38 are a few examples of such integration of methods. In one study, Dhar and Ranganathan39 compared the performance of KBS approaches with OR/MS methods in solving a large faculty-to-course assignment problem. They found that when constraints were not satis®ed, the integer program-
R Santhanam and J ElamÐA survey of knowledge-based systems research in decision sciences (1980±1995)
ming approach failed to ®nd a feasible solution even after many hours of execution. On the other hand, the KBS was able to provide a useful partial solution and indicate constraints that needed to be amended. Similarly, MS/OR methods can be used to improve the performance of a KBS. For example, Singh and Cook40 propose the use of an inference engine in a KBS that can optimise its search strategy. Therefore, research clearly indicates the value of integration of AI and MS/OR methods in development and use of a KBS. KBS technology and research can increase the productivity of the MS/OR practitioner and his or her ability to solve practical problems.41 Using several examples, Silverman42 identi®es speci®c research topics that can foster greater integration between KBS and MS/OR. Validation Validation is the process of comparing the output of the KBS with that of experts to determine the reliability and usability of the system. O'Leary43 provides a comprehensive framework and set of guidelines that can be applied by practicing knowledge engineers to validate a KBS. These are based on the principles of research design, and include the gauging of content validity, criterion validity, and systematic errors made by the system. Back44 utilises these guidelines and demonstrates how it was applied to a KBS for ®nancial system planning. King and Phythian,45 Kobbacy et al,46 and Sturman and Milkovich47 also provide case studies detailing the problems faced in validating a KBS. King and Phythian45 make a distinction between validation procedures for a KBS that operate in a formal domain (where knowledge is clearly de®ned and openly examinable) vs a non-formal domain (where knowledge is socially constructed and there are many views of the truth). Validation is more dif®cult in the latter situation. They provide a validation framework that is applied to an ES for tender inquiry evaluation. Kobbacy et al 46 examined the validation of a KBS that optimises the maintenance activities of large technical systems such as telecommunication systems. They found that the validation exercise highlighted the need for good test cases that examine all aspects of the system's problem solving abilities. During the validation process of an ES for choosing ¯exible employee bene®ts, Sturman and Milkovich47 found that the system passed several tests of construct and criterion validity. However, they discovered that in certain types of bene®ts, there was a low degree of difference between novice and expert judgment suggesting that the system did not improve upon the decisions made by the novice user. This suggests that knowledge engineers should not test the system for just a few cases but they should test the system for all the different types/classes of decisions that it provides support for. This will better ensure the reliability and value of the system. The researchers also found that in
451
many decision situations, there is a low level of agreement among experts on the right solution, which can be problematic in a validation procedure. O'Leary48 provides methods to determine the extent to which multiple experts used in the validation exercise make similar judgments. This will provide useful information when embarking on a validation exercise. Management Research ®ndings thus far indicate that the successful management of a large scale KBS is in many ways similar, but in substantive ways, different from those for large scale, traditional computer-based systems.49 Research in this category highlight these similarities and differences by drawing on the actual experiences involved in designing, developing, and implementing KBS in various organisations. The most commonly proposed model dealing with the management of KBS parallels the model proposed in research dealing with the design and development process. It consists of ®ve basic stages: Identi®cation, Conceptualisation, Formalisation, Implementation, and Validation.50 Important aspects to be that have to be managed in each of these stages are discussed. Identi®cationÐThe identi®cation stage involves the selection of a domain, a problem within that domain, and a particular task related to that problem that a KBS will perform. It is suggested that in this stage the goals of the system, its associated bene®ts, the required performance criteria and resources required for development is also clearly identi®ed. It is extremely important to choose a problem that is amenable to a KBS solution and is also recognised as important by the ultimate system user.51,52 Some of the approaches found useful for identifying speci®c KBS applications in an organisation included the use of a task force composed of people from the knowledge engineering group, experts, and user management53 and the establishment of a steering committee made up of strategically focused managers representing various business functions.49 A set of guidelines to determine the problem complexity, in terms of the technology required and the depth of knowledge to be embedded in the system are described by Meyer and Curley.54 Practitioners can use this framework to determine the problem complexity which in turn will inform them on various aspects of the project effort required such as the development time required, extent of staf®ng, and the extent of organisational support needed. Compared to traditional IS projects, the new dimension of knowledge engineering in KBS projects requires attention to two new roles in the project team. These are the domain expert and knowledge engineer. It is extremely important to make certain that a domain expert is available and willing,50,55 has knowledge of computers and good
452 Journal of the Operational Research Society Vol. 49, No. 5
communication abilities. It is preferable that this expert is acceptable to the ultimate end user.56 The desirable characteristics for a knowledge engineer include the following: analytical abilities, an ability to develop rapport with an expert, curiosity, unconventional educational backgrounds, and conventional programming knowledge.53 Since it may be dif®cult to ®nd individuals with these characteristics within an IS department, it is suggested that knowledge engineers be speci®cally trained to develop these skills.50 However, ®eld surveys57 indicate that many practising knowledge engineers feel that they were trained on technical skills but not on interpersonal and organisational skills. A successful training program in a large corporation has been described in Maletz58 that informs managers on how to set up training programs for knowledge engineers. ConceptualisationÐManagers should be aware that this stage is often dif®cult and will consume a large part of the project effort. This step is primarily concerned with knowledge acquisition and knowledge structuring. Many different approaches have been proposed for accomplishing the knowledge acquisition phase including observation of the expert, interviewing, using structured question and answering sessions, analysing verbal protocols and using manuals, etc. The important fact to be remembered is that no one method may be suf®cient and therefore organisations are urged to consider alternative knowledge acquisition approaches before starting a KBS project.59 FormalisationÐThis stage is concerned with formally representing the knowledge of the expert elicited during the previous stage. Organisations are advised to choose an appropriate knowledge representation language that is rich enough to describe the domain of the system and is compatible with an expert's knowledge. This could determine the success or failure of a particular KBS project.55 ImplementationÐImplementation involves the building of a prototype of the KBS. The use of rapid prototyping and interactive testing prototypes is strongly recommended in KBS development51,60 since it has been found to be extremely valuable in rethinking the problem and redesigning the system.61 Organisations should expect this stage to be tedious, time consuming, iterative, and incremental. A major issue to be addressed during this stage is the decision concerning the appropriate tools to use in building the prototype. It is clear that all tools have limitations and developers need to learn how to creatively overcome these limitations.62,63 Implementation also encompasses those activities that are required to turn the prototype into a production system utilised by a large number of users. Findings from case studies and ®eld surveys of KBS indicate that as for any information system, user involvement in the project, a skilled project team, management support, the interface
characteristics of the shell and end-user support centers greatly facilitate the implementation process by increasing satisfaction by end-users.52,64,65 Important issues other than system design and development to be addressed during implementation include such things as whether the use of the KBS should be discretionary or mandated;60 how to ensure that the system gains acceptance of the end user;55 how to integrate KBS into the users' work processes,62 and planning for the changes that are likely to result from the implementation of the system. Yoon et al 52 and Yoon and Guimaraes56 reported that end-users seem to be most satis®ed and readily accept systems that increase the importance of their job, increase their freedom to do their job, and provide opportunities for advancement. It is also important that managers recognise that the system will require resources and mechanisms for ongoing updates even after successful implementation.55 For example, it is estimated that about 40% of the rules in the XCON knowledge base change each year.63
Use and impact Our survey results indicate that KBS technology has been accepted as a tool to improve decision making in a variety of business applications. The majority of them are in production operations66 and in ®nancial planning problems.67,68 A complete listing of the various applications for which KBS have been proposed is given in Wong and Monaco.69,70 While many of the proposed systems are laboratory systems, there are several case studies of actual deployment of KBS in ®eld settings.71,72 Based on these results, it is very clear that managers seeking to adopt this technology for their organisation can do so knowing that this technology has proven to be useful in many public and private corporations, and in large and small business organisations. The extent of usage or diffusion of this technology in corporations however is not very clear. One survey73 compared the extent of use of KBS in business organisations in Singapore and USA and concluded that USA is much more ahead in adopting KBS technology. A survey undertaken among organisational members of OR Society in United Kingdom found that about 35% of UK organisations are involved in developing and using KBS technology.74 Surveys indicate that the most successfully used KBS are those that address the core of the business operations rather than functions on the periphery.75 Perhaps that is the reason why certain large corporations such as IBM, Digital Equipment Corporation, Coopers and Lybrand and American Express, actively develop KBS and tie it with their key business operations.76 It was found that one third of the early systems such as Expertax, Authorizers Assistant, and XCON, etc. developed during the early 1980s are still thriving. Managers should be aware that falling into disuse
R Santhanam and J ElamÐA survey of knowledge-based systems research in decision sciences (1980±1995)
of the remaining systems were attributed to organisational issues, such as lack of user acceptance, problems in the transition from development to maintenance, inability to maintain developers and shifts in organisational priorities, rather than being attributed to technical issues.76 Another problem stalling the actual deployment of ES in some cases could be the legal implications or liability of using these systems.77 Some systems have been abandoned because the developers feared legal liability in case of poor advice.76 Mykytyn et al77 state that in the future, organisations, knowledge engineers and experts may be held liable for injuries resulting from faulty advice given by a KBS. The authors provide a framework and some checks and actions that can be administered by organisations to protect themselves from costly litigation. In one industry, namely healthcare/medicine, many prototype systems have been developed. However a survey of the status of use of these systems shows that very few systems are used in actual ®eld diagnosis.78 One reason for disuse is that physicians and hospitals have to await resolution of social and legal issues before they would be committed to use of these systems. Another problem limiting the use of medical systems is the dif®culty of totally automating the complexity of an expert medical decision process. Physicians are not fully convinced that a system can `replicate eight years of medical education and training and countless hours of real world experience'.78 It is perhaps for these reasons that some researchers feel that the greatest use of AI technology will not be systems that replace human judgment or decisions, but those that assist decision making. Therefore, they argue for KBS systems that are extensions of DSS rather than full-¯edged ES.79,80 Some empirical support for this suggestion is provided in Grabowski and Wallace.81 In this study, the authors tested an expert system developed for shipboard navigation. They found that the system did not improve performance on a task called track keeping. This was attributed to the fact that the ES could automate the cognitive skills used by experts in piloting vessels but not the motor and perceptual skills required. The authors believe that experts often use perceptual skills, for example pattern recognition in ®nancial data, which may not be easily automated. The implication is that managers have to closely study the task involved to see the type of skill required to accomplish the task. For purely cognitive skills, current technology can help to automate it. If many perceptual skills are required, it may not be easily automated. In this situation, decision aiding systems that improve the decision may be a fruitful option. It is to be noted that decision aiding systems typically use operations research models and KBS technology suggesting greater opportunities to integrate the two technologies. There are very few studies that have rigorously attempted to study the actual bene®ts from that have accrued to an organisation through the use of a KBS. The study by Sviokla82 uses a case study approach to gauge the
453
impact of the expert system, XCON, an expert system that con®gures VAX and PDP computers at Digital Equipment Corporation (DEC). The impact of XCON was examined by comparing performance programs and information processing capacity of DEC between two periods, namely pre-XCON (1978) and post-XCON (1985). The results showed a signi®cant improvement in the quantity of the output, quality of the output documents, the number of users and the ef®ciency of the con®guration process. In addition, to the above, organisational changes occurred and roles and responsibilities of people associated with the con®guration task became different. People supporting the development and use of XCON assumed greater responsibility while the technical editor's job became more clerical. Maintenance of the systems knowledge base became more important and new people were entrusted with this responsibility.
Discussion Our survey results indicate that research in KBS for the last ®fteen years has already provided a rich body of knowledge that can direct future research and practice. The large number of papers that have been published and the variety of applications that have been addressed supports the notion echoed by other researchers that KBS is fast becoming a reference discipline for DSS research.3 Our research indicates that KBS has been most applied to address production and operations problems, to ®nancial decision making situations and to computer-related applications. This is understandable because the very nature of KBS technology makes it easier to apply it to very structured problems such as those found in production and operations management. Secondly, cost savings due to the use of KBS technology in these areas is probably more clearly identi®ed and thus become good candidates for applying this technology. Research in decision sciences has to eventually lead to practical applications. This thrust is evidenced by the large number of papers providing proposals for system development. The fact that design proposals represent a very high percentage of article-type indicates that there are fewer experiments and survey type research being conducted. Therefore, the design principles and feasibility of building a KBS are clear but what is less clear is how these systems affect the actual quality of decision making. What problems do decision makers encounter when they have to use a KBS? What evidence is available to support the idea that the decision process is enhanced with use of KBS tools? In the DSS literature several studies83 have been conducted to look at the impact that DSS have on decision making strategies and DSS effectiveness. During the survey period, no such study seems to have been conducted with KBS. Similarly, from a macro perspective there are very few
454 Journal of the Operational Research Society Vol. 49, No. 5
studies that examine the impact of using KBS in organisations. Our survey results support recent suggestions made by other researchers.2,84 Will and Hardaway84 showed that much of the advice suggested for developing KBS is contradictory. Therefore, they call for more rigorous investigations into research studies. Research that employs ®eld surveys, case studies and experimental methods are needed to identify the context in which the given advice is applicable and why some systems succeed, and others fail. These other research methods have to be employed to understand the in¯uence and bene®ts that organisations are obtaining from deploying KBS. The results might also provide information that can lead to further research on how to design more effective KBS. In terms of research topic addressed, smaller number of papers in the management and validation classi®cation suggest that these areas are not well researched. As KBS becomes more prevalent in organisations, much more research in this area will be needed. Several issues emerge under each of the research topics. Research on the design and development topic show that building KBS is a useful and feasible proposition but several issues relating to design need to be examined further. Despite the fact that correct problem identi®cation is a correlate of ES success,52 there are not many guidelines or tools that have been developed to guide practicing managers to discriminate between a good and bad candidate for KBS application. We found only three papers that speci®cally dealt with the issue of identifying the appropriate problem. To select KBS tools, several techniques and systems21,85 have been suggested to help developers choose the most appropriate tool. Similar research on the development of a framework or tool that can suggest whether a task can be automated or not will be useful. Since research indicates that managerial issues are also critical to success, the tool should test problem characteristics and organisational characteristics to decide on whether the task will bene®t from a KBS. Knowledge acquisition is a bottleneck in ES and ruleinduction methods have been suggested as an alternative. It is not clear as to which of the methods is useful in a speci®c domain. Future research can address this issue and provide some guidelines on how to choose an appropriate method or how to combine knowledge acquisition methods in a speci®c problem situation. Problem complexity has been identi®ed as an important variable affecting the development success of an ES. How can this issue be tested and applied to design decisions, such as the choice of a tool or method of development? Research results indicate that the interface and explanations of a KBS have to be tailored to the user's skills and problem solving strategies. How does one determine the skill level of the user and his or her approach to problem solving? Some researchers even argue that the interface
should also have some knowledge about the user and his or her knowledge level so that the advice can be tailored to their requirements.86,87 How can this be accomplished practically? Given that KBS are used by a variety of users, would this be a feasible approach? Can design guidelines that allow a builder to make easy-to-use interfaces be applied to the development of KBS? Future research in these areas will enhance user acceptance of KBS. The value of integration of AI and OR methods has been proven. Research has to identify applications where these integration approaches are most useful and where it cannot be applied. To make this integration practical, are new kinds of tools, programming and interface technology needed? What is the difference in outcome when a mathematical model is combined with a KBS? Do managers prefer such mergers and do they make better decisions when these are merged? Future research can also address issues on how to validate the output of systems that combine mathematical models and qualitative reasoning. Validation of outputs has been recognised as an important aspect of KBS development. Research thus far has highlighted problems in validating a KBS, such as, lack of expert agreement on test cases, lack of appropriate test cases, and existence of problem situations where there are not many differences between an expert and novice judgment. Future research can address issues such as how to develop/obtain appropriate test cases. How do we alleviate problems resulting from differing expert judgments? What types of domain face greater dif®culties in validation and how can they be addressed? Most of the validation procedures suggest the use of the Turing test, that is, compare the system's output to that of an expert to see if there is an agreement. Since many systems are decision aiding systems, perhaps measures can be developed to determine the extent to which the system improves upon the decision making abilities of the user. The system can be tested to see if it provides a certain acceptable level of improvement in decision making rather than ensuring that it totally concurs with that of an expert judgment. Research has provided a lot of prescriptive information on the important issues in managing a KBS project. These have to be ®eld tested. Research on implementation success has primarily focused on the user acceptance and satisfaction with the system. DeLone and McLean88 show that success of any information can be gauged with different measures including frequency of use of the system, regularity of use, improvement in quality of decisions made, etc. Future research can determine how factors such as, user involvement, project staf®ng and top management support affect these other variables used to measure the success of a system. Similarly, ways to measure implementation success in terms of project development success have to be developed. Was the project a success in terms of resources consumed, delivery time, completion, etc.? Project manage-
R Santhanam and J ElamÐA survey of knowledge-based systems research in decision sciences (1980±1995)
455
Figure 1 The development and use of KBS.
ment literature provides many guidelines on how to manage a project and measure its success. How can they be applied to a KBS project? What are special problems relating to the acceptance and implementation of KBS? How should the development team be chosen and how should they be managed? What should be the skill level of members and how should they be trained? What are special issues relating to the maintenance of these systems? These and other questions have to be addressed and researched. Research from papers classi®ed under `Use and Impact' suggests that the best use of KBS is obtained when it is linked with business operations and goals. Research has to determine ways in which organisations can predict and measure the bene®ts of their investment in this technology. The effect and special needs of small businesses for KBS have been very sparsely researched. Research can also compare projects that failed in an organisation with those that succeeded to get a more detailed understanding of contextual factors affecting the eventual success of a KBS project. Similarly, inter-organisational research can identify industry factors that motivate organisations such as, Coopers and Lybrand and American Express to actively persist in developing KBS while it forces organisations such as Texas Instruments, and Xerox to refocus their efforts in other forms of technology.76
The number of published papers clearly indicates a high level of research interest in KBS in the last ®fteen years. In Figure 1 we highlight the key ®ndings of our survey. It also indicates some of the issues that need to be researched. Importantly, more empirical testing and ®eld surveys are needed to understand the value of these systems, the impact on individual and organisational decision making, the facilitating factors and barriers to utilising KBS in business organisations. AcknowledgementsÐWe thank Gary Walter for helping us with data collection on an earlier version of the paper. We also thank Chitra Solomon for assisting us with data collection in this version of the paper.
References 1 Hayes-Roth F and Jacobstein N (1994). The state of knowledge-based systems. Communications of the ACM 37: 27±39. 2 Henderson JC (1987). Finding synergy between decision support systems and expert systems research. Decis Sci 18: 333±343. 3 Goul M, Henderson JC and Tonge FM (1992). The emergence of arti®cial intelligence as a reference discipline for decision support systems research. Decis Sci 23: 1263±1276. 4 Dunn DW and Silverman BG (1995). Technology and intelligent decision support. Eur J Op Res 84: 1±4.
456 Journal of the Operational Research Society Vol. 49, No. 5
5 Elam J, Huber G and Hurt M (1996). An examination of the DSS literature (1975±1985) in decision support systems. In: McClean ER and Sol HG (eds). A Decade in Perspective, Proceedings of the IFIP Conference. Elsevier Science Publishers B.V.: New York. 6 Sviokla JJ (1986). Business implications of knowledge-based systems. Data Base 18: 5±16. 7 Jain HK and Chaturvedi AR (1989). Expert system problem selection: a domain characteristics approach. Information & Management 17: 245±253. 8 Stein E (1992). A method to identify candidates for knowledge acquisition. J of MIS 9: 161±178. 9 Agarwal R and Tanniru MR (1990). Knowledge acquisition using structured interviewing: an empirical investigation. J Mgmt Information Sys 7: 123±140. 10 Seagle JP and Duchesi P (1995). Acquiring expert rules with the aid of decision tables. Eur J Opl Res 84: 150±162. 11 Byrd A, Cossick KL and Zmud RW (1992). A synthesis of research on requirements analysis and knowledge acquisition techniques. MIS Q 16: 117±138. 12 Gregory H (1995). Soft systems models for knowledge elicitation and representation. J Opl Res 46: 562±578. 13 Braun H and Chandler JS (1987). Predicting stock market behavior through rule induction: an application of the learning-from-example approach. Decis Sci 18: 415±429. 14 Mingers J (1986). Expert systemsÐexperiments with rule induction. J Opl Res Soc 37: 1031±1037. 15 Tam KY (1990). Automated construction of knowledge-bases from examples. Information Systems Res 1: 144±167. 16 Liang P (1992). A composite approach to inducing knowledge for expert systems design. Mgmt Sci 38: 1±17. 17 Messier WF, Jr and Hansen JV (1988). Inducing rules for expert system development: an example using default and bankruptcy data. Mgmt Sci 34: 1402±1415. 18 Cronan P and Glorfeld LW (1992). Production system development for expert systems using a recursive partioning induction approach: an application to mortgage, commercial, and consumer lending. Decis Sci 22: 812±845. 19 Hu PMY and Hung MS (1993). Two-group classi®cation using neural networks. Decis Sci 24: 825±845. 20 Kimbrough SO and Lee RM (1988). Logic modeling. Decis Support Systems 4: 7±12. 21 Stylianou AC, Madey GR and Smith RD (1992). Selection criteria for expert system shells: a socio-technical framework. Communications of the ACM 35: 30±48. 22 Nelson W and Balachandra R (1991). Choosing the right expert system building approach. Decis Sci 22: 354±368. 23 Ye LR and Johnson PE (1995). The impact of explanation facilities on user acceptance of expert systems advice. MIS Q 19: 157±171. 24 Lamberti DM and Wallace WW (1990). Intelligent interface design: an empirical assessment of knowledge presentation in expert systems. MIS Q 14: 279±312. 25 Pei BK and Reneau JH (1990). The effects of memory structure on using rule-based expert systems for training: a framework and an empirical test. Decis Sci 21: 263±282. 26 Goul M and Tonge F (1987). project IPMA: applying decision support system design principles to building expert-based systems. Decis Sci 18: 448±467. 27 Doukidis GI (1988). Decision support system concepts in expert systems: an empirical study. Decis Support Systems 4: 345±354. 28 Connell NA and Powell PL (1990). A comparison of potential applications of expert systems and decision support systems. J Opl Res Soc 4: 431±439.
29 Elam JJ and Konsynski B (1987). Using arti®cial intelligence techniques to enhance the capabilities of model management systems. Decis Sci 18: 487±502. 30 Bharadwaj A, Choobineh J, Lo A and Shetty B (1992). Model management systems: a survey. Annals of Opns Res 38: 17±67. 31 Higa K, Morrison M, Morrison J and Sheng OR (1992). Object oriented methodology for knowledge base/database coupling. Communications of the ACM 35: 99±112. 32 Vassilou Y, Clifford J and Jarke M (1985). Access to speci®c declarative knowledge by expert systems: the impact of logic programming. Decis Support Systems 1: 123±141. 33 Phelps RI (1986). Arti®cial intelligenceÐan overview of similarities with OR. J Opl Res Soc 37: 13±20. 34 Glover F and Greenberg HJ (1989). New approaches for heuristic search: a bilateral linkage with arti®cial intelligence. Eur J Opl Res 39: 119±130. 35 Balestra G and Tsoukias A (1990). Multicriteria analysis represented by arti®cial intelligence techniques. J Opl Res Soc 41: 419±430. 36 Potvin JY, Lapalme G and Rousseau JM (1990). Integration of AI and OR techniques for computer-aided algorithmic design in the vehicle routing domain. J Opl Res Soc 41: 517±525. 37 Conrad M (1987). Rapprochement of arti®cial intelligence and dynamics. Eur J Opl Res 30: 280±290. 38 Liberatore J and Stylianou AC (1995). Expert support systems for new product development decision making: a modeling framework and applications. Mgmt Sci 41: 1296±1316. 39 Dhar V and Ranganathan N (1990). Integer programming vs expert systems: an experimental comparison. Communications of the ACM 33: 323±336. 40 Singh MG and Cook R (1985). A new class of intelligent knowledge-bsed systems with an optimization inference engine. Decis Support Systems 1: 299±312. 41 Zahedi F (1987). Arti®cial intelligence and the management science practitioner; the economics of expert systems and the contribution of MS/OR. Interfaces 17: 72±81. 42 Silverman G (1994). Unifying expert systems and the decision sciences. Opns Res 42: 393±413. 43 O'Leary DE (1987). Validation of expert systemsÐwith applications to auditing and accounting expert systems. Decis Sci 18: 468±486. 44 Back B (1993±1994). Validating an expert system for ®nancial statement plannning. J of MIS 10: 157±171. 45 King M and Phythian GJ (1992). Validating an expert support systems for tender enquiry evaluation: a case study. J Opl Res 43: 203±214. 46 Kobbacy A, Proudlove NC and Harper MA (1994). Towards an intelligent maintenance optimization system. J Opl Res 46: 831±853. 47 Sturman C and Milkovich GT (1995). Validating expert systems: a demonstration using personal choice expert, a ¯exible employee bene®t system. Decis Sci 26: 105±118. 48 O'Leary E (1993). Determining differences in expert judgment: implications for knowledge acquisition and validation. Decis Sci 24: 395±407. 49 Barker VE and O'Connor DE (1989). Expert systems for con®guration at digital: XCON and beyond. Communications of the ACM 32: 298±318. 50 Bobrow DG, Mittal S and Ste®k MJ (1986). Expert systems: perils and promise. Communications of the ACM 29: 880±894. 51 Goul M (1987). On building expert systems for strategic planners: a knowledge eingineer's experience. Information & Mgmt 12: 131±141. 52 Yoon Y, Guimares T and O'Neal Q (1995). Exploring the factors associated with expert systems success. MIS Q 19: 83±103.
R Santhanam and J ElamÐA survey of knowledge-based systems research in decision sciences (1980±1995)
53 Weitzel JR and Andrews KR (1988). A company/university joint venture to build a knowledge-based system. MIS Q 12: 23±36. 54 Meyer H and Curley KF (1991). An applied framework for classifying the complexity of knowledge-based systems. MIS Q 15: 455±472. 55 Bell MZ (1985). Why expert systems fail. J Opl Res Soc 36: 613±619. 56 Yoon Y and Guimares T (1995). Assessing expert systems impact on users' jobs. J of MIS 12: 225±249. 57 Mykytyn P, Mykytyn K and Raja MK (1994). Knowledge acquisition skills and traits: a self-assessment of knowledge engineers. Information & Mgmt 26: 103±104. 58 Maletz MC (1990). KBS circles: a technology transfer initiative that leverages Xeroxes leadership through quality program. MIS Q 14: 323±329. 59 Hershauer J, Karim A and Philippakis A (1989). A ®eld observation study of an expert system prototype development. Information & Mgmt 17: 107±116. 60 Braden B, Kanter J and Kopcso D (1989). Developing an expert systems strategy. MIS Q 13: 459±468. 61 Weitzel JR and Kerschberg L (1989). Developing knowledgebased systems: reorganizing the system development life cycle. Communications of the ACM 32: 482±490. 62 Leonard-Barton D (1987). The case for integrative innovation: an expert system at Digital. Sloan Mgmt Rev 29: 7±20. 63 Pollitzer E and Jenkins J (1985). Expert knowledge, expert systems and commercial interests. Omega 13: 407±418. 64 Agarwal R, Higgins C and Tanniru M (1991). Technology diffusion in centralized MIS environment. Information & Mgmt 20: 61±70. 65 Tyran K and George JF (1993). The implementation of expert systems: a survey of successful implementations. Data Base 24: 5±15. 66 Kusiak A and Chen M (1988). Expert systems for planning and scheduling manufacturing systems. Eur J Opl Res 34: 113±130. 67 Shane B, Fry M and Toro R (1987). The design of an investment portfolio selection decision support system using two expert systems and a consulting system. J of Mgmt Information Systems 3: 79±92. 68 Srinivasan V and Ruparel B (1990). CGX: an expert support system for credit granting. Eur J Opl Res 45: 293±308. 69 Wong K and Monaco JA (1995). A bibliography of expert system applications for business (1984±1992). Eur J Opl Res 85: 416±432. 70 Wong BK and Monaco JA (1995). Expert system applications in business; a review and analysis of the literature (1977±1993). Information and Mgmt 29: 141±152.
457
71 Kennedy DJ, Parnell GS and Rowell WF (1988). An expert system for scheduling satellite supports. Interfaces 18: 28±34. 72 Sullivan G and Fordyce K (1990). IBM Burlington's logistics management system. Interfaces 20: 43±64. 73 Thow-Yick L and Huu-Phuong T (1990). Management expert systems for competitive advantage in business. Information & Mgmt 18: 195±201. 74 Doukidis GI and Paul RJ (1990). A survey of the application of arti®cial intelligence techniques within the OR society. J Opl Res Soc 41: 363±375. 75 Meyer H and Curley KF (1991). Putting expert systems technology to work. Sloan Mgmt Rev 32: 21±31. 76 Gill G (1995). Early expert systems: where are they now? MIS Q 19: 51±81. 77 Mykytyn K, Mykytyn PP, Jr and Slinkman CW (1990). Expert systems: a question of liability. MIS Q 14: 27±42. 78 McCauley R and Ala M (1992). The use of expert systems in the healthcare industry. Information & Mgmt 22: 227±235. 79 Sheil B (1987). Thinking about arti®cial intelligence. Harvard Bus Rev 65: 91±97. 80 El-Najdawi MK and Stylianou AC (1993). Expert support systems: integrating AI technologies. Communications of the ACM 36: 54±65. 81 Grabowski M and Wallace WA (1993). An expert system for maritime pilots: its design and assessment using gaming. Mgmt Sci 39: 1506±1520. 82 Sviokla J (1990). An examination of the impact of expert systems on the ®rm: the case of XCON. MIS Q 16: 127±142. 83 Sharda R, Barr SH and McDonnell JC (1988). Decision support system effectiveness: a review and empirical test. Mgmt Sci 34: 139±159. 84 Will RP and Hardaway DE (1994). Advice for building expert systems: precepts or proverbs. Interfaces 24: 79±81. 85 Plant T and Salinas JS (1994). Expert systems shell benchmarks: the missing comparison factor. Information & Mgmt 27: 89±93. 86 Caroll JM and McKendree J (1987). Interface design issues for advice-giving expert systems. Communications of the ACM 30: 14±31. 87 Dos Santos BL and Holsapple CW (1989). A framework for designing adaptive DSS interfaces. Decis Support Systems 5: 1±11. 88 DeLone WH and McLean ER (1992). Information system success: the quest for the dependent variable. Information Systems Res 3: 60±95.
Received May 1995; accepted November 1997 after two revisions