Instr Sci (2009) 37:43–63 DOI 10.1007/s11251-008-9080-7
Web site credibility: Why do people believe what they believe? Marie K. Iding Æ Martha E. Crosby Æ Brent Auernheimer Æ E. Barbara Klemm
Received: 3 October 2008 / Accepted: 16 October 2008 / Published online: 27 November 2008 Springer Science+Business Media B.V. 2008
Abstract This research investigates university students’ determinations of credibility of information on Web sites, confidence in their determinations, and perceptions of Web site authors’ vested interests. In Study 1, university-level computer science and education students selected Web sites determined to be credible and Web sites that exemplified misrepresentations. Categorization of Web site credibility determinations indicated that the most frequently provided reasons associated with high credibility included information focus or relevance, educational focus, and name recognition. Reasons for knowing a Web site’s content is wrong included lack of corroboration with other information, information focus and bias. Vested interests associated with commercial Web sites were regarded with distrust and vested interests of educational Web sites were not. In Study 2, credibility determinations of university students enrolled in computer science courses were examined for 3 provided Web sites dealing with the same computer science topic. Reasons for determining Web site inaccuracy included own expertise, information corroboration, information design and bias. As in Study 1, commercial vested interests were negatively regarded in contrast to educational interests. Instructional implications and suggestions for further research are discussed. Keywords Web site credibility Web evaluation Critical information evaluation College students Credibility determinations Web site veracity
Preliminary results from Study 1 were presented as a poster entitled ‘‘Users’ Confidence Levels and Strategies for Determining Web Site Veracity’’ (Iding et al. 2002a) and appeared in associated proceedings for The WWW 2002: The Eleventh International World Wide Web Conference, in Honolulu, Hawaii. Preliminary results from Study 2 were presented as a paper entitled, ‘‘Judging the Veracity of Web Sites’’ (Crosby et al. 2002) and appeared in associated proceedings for the International Conference on Computers in Education (ICCE 2002) in Auckland, New Zealand. M. K. Iding (&) M. E. Crosby E. Barbara Klemm College of Education, University of Hawaii, 1776 University Ave, Honolulu, HI 96822, USA e-mail:
[email protected] B. Auernheimer California State University, Fresno, USA
123
44
M. K. Iding et al.
Introduction Since the advent of the World Wide Web, the general public has become increasingly reliant on the Web for information. Upon what bases do people evaluate information on the Web? How do students and others determine what is credible or scientifically accurate? Do they consider factors like commercial or vested interests of authors? What aspects contribute to their judgments? A recent article in Nature (Giles 2005) magazine highlights aspects of the issue by comparing the accuracy of science information in Wikipedia, a Web-based encyclopedia to which the general public can contribute, to that of Encyclopedia Britannica, long considered the accepted standard. Articles from both sources were sent to experts in their respective fields without identifying information sources. The experts then rated the accuracy of the encyclopedia entries. The following conclusion was published in Nature: ‘‘The exercise revealed numerous errors in both encyclopedias, but among the 42 entries tested, the difference in accuracy was not particularly great: the average science entry in Wikipedia contained around four inaccuracies; Britannica, about three’’ (p. 900). Despite the fact that Nature’s conclusion could be considered favorable to Wikipedia, Giles (2005) anecdotally highlights the controversial aspects of its open authorship process by describing the efforts of Wikipedia contributor Connolly, a British climate change researcher. Connolly’s descriptions of global warming were repeatedly edited and countered by ‘‘climate change skeptics.’’ Finally, Wikipedia administrators mediated the repeated editing and counter-editing by opposing contributors. Although the Nature article received wide press, the issue of information accuracy on the Web is of concern in many areas including medical, commercial, and educational realms. In this article, we define credibility, provide illustrative examples of research examining information accuracy or credibility judgments in these three areas, and present two studies in which university students from education and computer science classes evaluated the credibility of information on Web sites.
What is credibility? In work by Klemm et al. (2001), credibility is associated with information accuracy or veracity, and with reputation of Web site authors or institutional affiliations (Klemm et al. 2001). Fogg et al. (2002) provide a concurring definition of credibility in the context of Web site evaluation: Credible information is believable information….People perceive credibility by evaluating multiple dimensions simultaneously. In general, these dimensions can be categorized into two key components: trustworthiness and expertise. The trustworthiness component refers to the goodness or morality of the sources and can be described with terms such as well intentioned, truthful, or unbiased. The expertise component refers to perceived knowledge of the source and can be described with terms such as knowledgeable, reputable, or competent. People combine assessments of both trustworthiness and expertise to arrive at a final credibility perception (p. 9). In general, this definition provides a useful operational or working definition for credibility to serve as a foundation for examining credibility research in different disciplines.
123
Web site credibility
45
Health care and the Web An area of particular concern is the public’s acceptance of medical or health care information on the Web. To what extent do users accept as credible health care information they find on the Web? Benotsch et al. (2004) investigated adults with HIV and their determinations of Web site credibility. The authors describe positive aspects of obtaining medical information on-line: ‘‘For patients with chronic and life-threatening conditions, the Internet can serve as a source of hope, social support, and empowerment’’ (p. 1004). Further, the authors point out it can be a source of up-to-date treatment information. However, Benotsch et al. (2004) argue that the Digital Divide results in more educated persons of higher socioeconomic status (SES) having access to the information, and possessing reading skills and comprehension of basic medical terminology that enables them to more successfully evaluate the credibility of the information. In contrast, less educated persons of lower SES backgrounds are clearly at a disadvantage. In their study, they compared the credibility determinations of persons with HIV to those of medical professionals. The Web sites they used for evaluation included an article describing treatments for HIV from the Journal of the American Medical Association (JAMA) and a less reputable article from a site describing a supposed cure involving goat serum extraction. Results indicated that persons with HIV attributed higher credibility to both sites than did health care professionals. Furthermore, the more literate and knowledgeable participants (regarding HIV treatments) tended to rate the JAMA site more highly than did participants with less knowledge and lower literacy levels. As Benotsch et al. (2004) explain: The present findings…suggest that some patients do not always evaluate online information critically and may be vulnerable to misinformation. The nature of the AIDS epidemic in the United States is such that educationally and economically disadvantaged groups are increasingly affected by HIV. Such persons are among the least equipped to critically evaluate the information they receive on-line (p. 1009). Compounding the seriousness of these findings is the fact that participants with HIV trusted their physicians most highly as the source of information about health care, then they information from the Web secondly. The authors conclude that, ‘‘Individuals most in need of information concerning HIV—those who would most benefit from the opportunities the Internet affords for learning new information may also be the most vulnerable to misinformation or to unethical peddlers of sham cures sold via the Internet’’ (p. 1009). Kalichman et al. (2006) subsequently provided a 2 month intervention focusing on basic computer and ‘‘Internet information consumer skills’’ (p. 545) for persons with HIV/ AIDS. Participants learned critical information evaluation skills, and used the Internet more to locate health information and for social support than did the control support group. These findings indicate that instruction can be effective for those with particular healthcare information needs and specific vulnerabilities. In other research, Escoffery et al. (2005) found that 74% of the undergraduates they surveyed found health information on the Internet. While it may seem that undergraduates would be more similar to the more highly literate and educated participants in the Benotsch et al. (2004) study, other research by Metzger et al. (2003) indicates that although college students appear to be aware that information from the Web (not specifically health care information) may not be highly credible, they tend to verify it less than do members of the general public. It may be that this finding reflects different purposes for information
123
46
M. K. Iding et al.
retrieval. Perhaps college students finding information for course assignments are less likely to be concerned about credibility than about simple relevance; whereas it may be that more highly educated and literate people may engage in verification processes if the information is associated with high levels of importance and could be associated with potentially costly or disastrous outcomes if wrong, such as may be the case in deciding upon a health care treatment.
Consumers and the Web Another area in which people are vulnerable to inaccuracy or non-credibility of Web-based information is consumerism. In a large-scale study, Fogg et al. (2002) examined 2,684 consumers’ responses to credibility of Web sites in a number of content areas, including e-commerce, finance, health, entertainment, sports, travel, and news. They found that consumers’ reasons for making actual credibility determinations were different from what they said their reasons were for making these determinations (reasons were reported in an earlier study by Princeton Survey Research Associates 2002). The authors explain, ‘‘We found a mismatch, as in other areas of life, between what people say is important and what they actually do’’ (p. 6). For example, when asked generally what factors they would consider in making Web site credibility determinations, consumers listed considerations like the presence of a privacy policy. In actual practice, they found that people rarely if ever referred to these criteria in making determinations. Instead, the authors found that credibility judgments (in terms of reasons people gave for making their decisions) focused first upon ‘‘design look’’ (i.e., ‘‘elements of the visual design, including layout, typography, white space, images, color schemes’’ p. 24). Other factors that were commented upon by consumers most frequently included ‘‘information structure’’ (i.e., how the information was structured upon the Web site) and ‘‘focus’’ (i.e., depth and breadth of information). It appeared to the authors that there was a difference between making credibility judgments about information on the Web and ‘‘what people notice when they evaluate a Web site for credibility’’ (p. 7). Stanford et al. (2002) carried out another study, in which they examined credibility judgments of 15 experts in the areas of health and finance and compared their determinations to those of the consumers in the earlier study. They found that, in contrast to consumers, health experts rated the following factors most highly in determining Web site credibility: ‘‘name reputation of a site, its operator or its affiliates,’’ ‘‘information source,’’ or references and ‘‘company motive’’ (p. 4). Finance experts focused on ‘‘information focus,’’ ‘‘quantity of information,’’ and ‘‘company motive’’ (p. 4). Stanford et al. (2002) conclude by calling for research that examines credibility judgments in other content areas. They also propose that consumer education in credibility assessment is essential, a conclusion that would concur with educators’ perspectives as well. Flanagin and Metzger (2007) examined people’s credibility determinations in response to Web sites in e-commerce, special interest, news organizations, and personal Web sites. They found that personal Web sites were rated lowest and news sites the highest. Like Fogg et al. (2002), they found that design aspects of sites had a greater impact on credibility determinations than knowledge of Web site sponsors. Flanagin and Metzger (2007) also found a discrepancy between self-reports of behavior and actual behavior, with those reporting that they had engaged in extensive Web verification actually verifying information less than others.
123
Web site credibility
47
Work by Tormala and Petty (2004) highlights aspects of credibility of information sources, and while not dealing directly with Web-based information sources, applies directly to this topic. Their research is important because it addresses peoples’ certainty of their attitudes and the link to possible behavior after exposure to persuasive arguments from expert and non-expert sources. For example, in an experiment investigating participants’ responses to advertising for a fictional aspirin, participants received information about the product and listed their own counterarguments against it. Findings indicated that ‘‘Participants became more certain of their attitudes when they resisted persuasion from an expert source, as long as they had sufficient cognitive resources available (presumably to reflect on the implications of their resistance)’’ (p. 434). The same effect did not occur when the persuasion was associated with a non-expert source. In a second experiment involving a fictional proposed change to a university examination policy, they found that ‘‘participants’ attitudes became more predictive of behavioral intentions after they resisted a persuasive attack from an expert source’’ (p. 438). The implication is that people become more convinced that what they believe is true when they resist weak arguments from high credibility sources. This may have implications for those who design Web sites and the degree of thoughtfulness with which they present information on those sites.
Research in education One of the general themes emerging from all of this research is the need for relevant education. Education can take the form of providing guidelines on the Web for evaluating Web-based information in particular areas, or in instructional interventions. In addition to educating k-12 and college students, it is also important to work with pre-service and practicing teachers who will be responsible for facilitating critical credibility determination skills in their students. Klemm et al. (2001) were particularly interested in how scientists and pre-service teachers evaluated information sources. At that time, the authors’ interest was in the credibility of sources generally, not only in Web-based resources, because of the current trend in k-12 science instruction away from textbook-based instruction and toward more constructivist modes that incorporate multiple resources. It appeared that the lack of refereeing of many of these sources left credibility judgments up to teachers and students themselves, more so than ever before, when textbooks were assumed to be definitive sources upon which entire curricula were based. Klemm et al. (2001) investigated credibility associated with a whole range of information sources (31) that might be utilized by teachers and others including tradebooks, CNN, TV newsmagazines like 20/20, a scientist working on a research question, and information from the World Wide Web. In order to provide a more veridical task, a context for credibility determinations was provided: the area of assessment of risks. Results indicated that pre-service teachers and scientists differed on assessing the credibility of the majority of the information sources. Elementary pre-service teachers (with fewer required science courses) were less like scientists in their determinations than secondary science pre-service teachers. To illustrate, scientists selected ‘‘scientist researching the risk topic in question’’ as the highest in credibility; secondary teachers rated the same category as fourth in credibility, and elementary pre-service teachers rated this category as eleventh in credibility. Additionally, pre-service elementary teachers selected the following two categories as having the highest levels of credibility: ‘‘popularized science magazines (e.g., Discover)’’
123
48
M. K. Iding et al.
and ‘‘resources from a museum, aquarium, or nature center.’’ Other information sources that were rated by elementary pre-service teachers as having high levels of credibility included ‘‘CNN Cable News Network,’’ ‘‘weekly newsmagazines (e.g., Times, Newsweek, etc.),’’ and ‘‘TV News Magazine (e.g., 20/20 or Dateline).’’ This high regard for the credibility of the popular media was not shared by scientists, who gave such programs as ‘‘TV News Magazines’’ very low ratings. As a scientist explained, ‘‘I place the most confidence in individuals with good judgment and broad experience whom I am acquainted with. This option isn’t listed. I have the least confidence in rapid turn-around sources (e.g., local television and newspapers that demonstrate a consistent lack of judgment and experience’’ (p. 89). Despite disagreements about the credibility of popular media, there were some sources of concurrence. For example, all three groups rated the following sources highly: ‘‘resources from university cooperative extensions (e.g., Sea Grant, Coop. Agriculture, etc.’’ and ‘‘Resources from a museum, aquarium, or nature center.’’ From an educational perspective this is encouraging, yet the research indicated a great need for education in informed credibility judgments for educators, the general public, as well as students in all areas. In response to the need for education in this area, Iding et al. (2002b) investigated the effects of instruction upon credibility judgments. For example, in an initial instructionoriented study, they worked with high school seniors enrolled in two biology classes over 4 days. The authors were interested both in factors contributing to high school students’ judgments as well as the effects of instruction on their ability to make informed decisions about the credibility of Web sites and the information contained in Web sites. As a pretest, the authors asked students to describe characteristics that they consider when deciding to use a Web site generally, then when deciding to evaluate scientific information on a Web site. Next, instruction was provided in three aspects of Web evaluation, including ‘‘credibility of authors and institutions, validity or accuracy of information…and presentation aspects…of the Web site and its information’’ (p. 376). These characteristics were adapted from Rader (1998) and Farah (1995) and used in the work of Nguyen (2000). Students’ lists of criteria were more complete after instruction, and ‘‘the majority of students reported learning something new, indicated that they would spend more time evaluating scientific information on Web sites, and reported increased confidence in their ability to evaluate scientific information on the Web’’ (p. 373). Iding et al. (2002b) concluded that it is possible, in a relatively short amount of time, to impact students’ Web assessment skills in a positive fashion. However, the authors were aware of the brevity of this instruction and the need for a longer course of instruction and follow-up evaluation to determine whether long-term transfer would be affected. In yet another study, Iding and Klemm (2005) worked with pre-service teachers who identified criteria that they used for determining whether they would cite scientific information from a Web site, critically evaluated actual scientific Web sites, then created structured evaluation forms (i.e., rubrics) for their students to use in critical Web site information evaluation. Students did not agree on the credibility of the science Web sites, even the NASA site.
The present studies All of the reviewed research underscores the need for effective Web-based information credibility determination skills among the general public. The issue is particularly important, as with the exponential proliferation of information on any topic on the Web,
123
Web site credibility
49
people become the arbiters of information accuracy—in domains in which they are knowledgeable and in domains in which they are not. How do people make these determinations? Do they consider potential biases and vested interests of Web site authors as they determine whether information is acceptable and accurate, believable, or not? The studies that follow address these questions. Specifically, the credibility determinations of university students in different disciplines, education, and computer science, are investigated as they make determinations of credibility of various Web sites related to their disciplines, rate confidence in their own determinations and articulate Web site authors’ vested interests.
Study 1 In the first study, the authors investigated credibility judgments of two groups of university students: computer science students and education students. The authors were interested in students’ judgments of characteristics they associated with credible Web sites and Web sites they determined to be non-credible. The authors were also interested in the students’ confidence in their own credibility judgments and in their determinations of Web site authors’ vested interests. Method Participants Participants consisted of 84 university students. Forty-seven students were enrolled in a computer science class, 21 in an educational psychology class, and 16 in a science methods class. Materials and procedure One of the authors, a computer science instructor, developed an exercise related to Web site selection and evaluation that was relevant to a topic covered in his class, cleanroom software engineering. The exercise was presented on a worksheet. In the exercise, students collaborated in small groups to select Web sites relevant to the topic. They were instructed to ‘‘Select a Web site that your group feels gives the most accurate and objective representation of cleanroom software engineering.’’ They explained why they selected that site, and described the author’s vested interests. They were also instructed to ‘‘Select a Web page that your group feels illustrates a misconception (or leads to a misconception) about cleanroom software engineering.’’ They explained the following: why this site illustrates a misconception; whether they thought the misconception was due to a deliberate attempt to mislead or to a mistake; and, what they believed this author’s vested interests were. In the next section of the exercise, they were asked, ‘‘How do you know when a Web site’s content is wrong?’’ Next, they were asked to answer the following questions as a group: On a scale of 1 (no confidence) to 5 (complete confidence), how confident are you about detecting misrepresentations on Web sites in general? On the same 1–5 scale, how confident are you in detecting misrepresentations in the particular Web site you [selected earlier].
123
50
M. K. Iding et al.
Finally, each student answered the following questions individually: On a scale of 1 (no competence) to 5 (complete competence) scale, how competent are you in evaluating the validity of information about cleanroom software engineering? On the same 1–5 scale, how competent are you in evaluating the validity of information on the Web in general? The exercise was modified to be content-appropriate for the education classes, so that it focused on science, technology and society (STS) for the science methods class and collaborative teaching project topics (such as motivation and multicultural education) for the educational psychology class. In the educational psychology class, instructions were: ‘‘Your task is to find information about the topic that your group has selected for the inclass cooperative teaching project.’’ Examples of topics included classroom management, special education, multicultural education, and motivation. Students were instructed to ‘‘Select a WWW page that your group feels gives the most objective and accurate representation of your topic.’’ They were also instructed to ‘‘Select a Web page that your group feels illustrates a misconception (or leads to a misconception) about your topic.’’ For the science methods class, students were instructed to ‘‘Select a WWW page that your group feels gives the most objective and accurate representation of your STS topic.’’ Examples of topics included ‘‘whale migration and environmental influences’’ and ‘‘genetic profiles.’’ As in the other groups, students were instructed to ‘‘Select a Web page that your group feels illustrates a misconception (or leads to a misconception) about the STS topic.’’ All other questions about reasons for accuracy determinations, vested interests, confidence and competence were the same for these students as those asked of the computer science students. Coding Two of the authors collaboratively coded students’ written comments in response to each question. Each separate statement that represented a major idea was coded, and frequencies were calculated for each category. The coding scheme serving as the basis for content analysis consisted of categories that emerged from the data and categories that were adapted from the work of Fogg et al. (2002). (Categories adapted from the work of Fogg et al. 2002, are indicated in Tables 1, 3, and 5). Results and discussion The authors coded participants’ responses to the question, ‘‘Why did you pick this Web page?’’ (See Table 1 for categories and results). Notably, the primary reason was information focus, or relevance. As one group explained, ‘‘Because it gives you the right information.’’ Another wrote, ‘‘Gave a detailed description of the cleanroom approach.’’ Other comments indicated that the amount of information affected their choice: ‘‘Extensive selection of previously done cleanroom projects.’’ ‘‘It had detailed descriptions and history of cleanroom software engineering.’’ Secondly, participants were interested in Web sites that appeared to be focused on providing educational information, or having an educational purpose. Name recognition was also important, as one group explained, ‘‘Carnegie Mellon is a famous educational institution.’’ Another wrote, ‘‘It is made by IBM and we figured that since they are such a successful company that they would be a creditable [sic] source.’’ Other reasons included references used in the site, information design, and design look. To differentiate between information design and design look, comments like the following
123
Web site credibility
51
Table 1 Study 1: Participants’ responses to question: Why did you pick this Web page? Categories
Frequency
Information focusa
27
Education
10
Name recognitiona
8
Links
6
Commercial interest/bias
5
Reference
4
Information designa
4
Currency of informationa
4
Design looka
3
Expertise
2
Biasa
2
Corroboration
1
Other
1
a
Indicates categories adapted from Fogg et al. (2002)
characterized information design, ‘‘The design had serious thought behind it which I credited to the Author’s over all effectiveness presenting the topic.’’ Information look was reflected in comments like, ‘‘Looks well set out.’’ Additional reasons are listed above are self-explanatory. Bias was used to characterize comments such as ‘‘Also discusses possible ‘down-sides’ to the technique.’’ Finally, a comment indicating corroboration referred to ‘‘external reviewers from Lockheed Martin.’’ In the next question, participants were asked, ‘‘What are the Web site authors ‘vested interests’?’’ Their responses are summarized in Table 2. Since participants were searching for objective and accurate Web sites, it is unsurprising that they described educational vested interests most frequently. The commercial vested interests of some Web site authors, the second most frequently described category was unexpected in association with these highly credible Web sites, as participants expressed general suspicion of the motives/ accuracy of commercial Web sites in other parts of the exercise. In the next question, participants were asked, ‘‘How do you know?’’ These answers (in Table 3) appeared to be more indicative of participants’ reasoning processes in determining Web site authors’ vested interests. We found that students were most likely to detect commercial interests of authors. This was shown in comments like, ‘‘There is a link to their products and services….’’ Table 2 Study 1: Participants’ responses to question: What are the Web site authors’ vested interests? Categories
Frequency
Education
14
Commercial
9
Other
5
Persuasion
3
Research and development
2
Name recognition*/promotion
1
*Indicates categories adapted from Fogg et al. (2002)
123
52
M. K. Iding et al.
Table 3 Study 1: Participants’ responses to question: How do respondents know about Web site authors’ vested interests? Categories
Frequency
Commercial
7
Education
5
Personal testimony
5
Affiliation
5
Name recognition/reputationa
3
Links
3
Other
3
Persuasion
2
Information biasa or unbiased
2
Information designa
2
a
Indicates categories adapted from Fogg et al. (2002)
Secondly, they mentioned indications of educational vested interests. These comments were especially interesting, as some participants appeared to associate educational motives with absence of vested interest, or what appeared to be objectivity or absence of bias. As one group stated, when asked what the author’s vested interests were, ‘‘None? It’s an educational source—ERIC.’’ In response to the next question, they said, ‘‘because it say [sic] it’s from ERIC Clearinghouse on Urban Education.’’ Another group stated, ‘‘It appears that the authors’ do not have some alterior [sic] motive behind their presentation, but instead are just trying to educate people on the cleanroom software process.’’ When asked, ‘‘How do you know?’’ they elaborated, revealing a negative bias against sales interests, ‘‘First off, there are no banners or pop-up ads. This is usually a big clue that the presenter doesn’t really care about the content and is just trying to sell you something.’’ It seems that educational vested interests were regarded as acceptable, while commercial-related vested interests were regarded with suspicion. The suspicion related to commercial interests seemed understandable but the unquestioning acceptance of educational interests is a promising area for further research. Participants seemed to not consider possible persuasion and/or political vested interests that can underlie educational Web sites. However, participants could have been more likely to have demonstrated awareness of persuasion or political vested interests had we selected controversial topics for them to examine. Participants also mentioned authors’ motives in sharing their own experiences with the cleanroom technique, i.e., personal testimony, and mentioned other aspects that would typically be associated an expression of vested interests (e.g., persuasion, and name recognition/reputation). They commented on whether authors appeared to provide a balanced presentation of benefits and drawbacks to the cleanroom technique or a one-sided perspective. Some participants examined links to other sites as well. Finally our participants (of a notably smaller sample) were less likely than participants in the Fogg et al. (2002) study, to mention ‘‘design look,’’ which in the present study was referred to as ‘‘information design.’’ Participants also responded to the question, ‘‘How do you know when a Web site’s content is wrong?’’ (See Table 4). The category of responses with the highest frequency was, ‘‘corroboration,’’ or the tendency of the Web site’s information to contradict users’ knowledge about the topic or to not agree with other known information accepted as valid
123
Web site credibility
53
Table 4 Study 1: Participants’ responses to question: How do you know when a Web site’s content is wrong? Categories
Frequency
Corroboration
13
Information focusa
10
Biasa
8
Expertise
6
Reference
6
Information designa
4
Name recognitiona/affiliation
4
Inaccuracy
3
Currency of informationa
2
Information claritya
2
Commercial interest
2
Tonea
2
Other links
1
Other
1
a
Indicates categories adapted from Fogg et al. (2002)
(e.g., Web sites, etc.). This is reflected in comments like, ‘‘We try to verify it with another unbiased source.’’ ‘‘You test what they are saying against a source you are sure is creditable [sic] and compare.’’ ‘‘Generally the validity of a Web site’s content comes into question when the design personal knowledge/experience contradicts what is being presented on the Web site….’’ Information focus was also associated with discrediting a Web site’s information. This included insufficient information: ‘‘They don’t have a lot of facts to back up the information.’’ An awareness of bias was reflected in comments like, ‘‘When the Web sites seems very narrow minded and only seem to present one way to do it and that must be the right way.’’ Confidence and competence in Web site evaluations Students provided self-ratings in the following areas, which were generally high: confidence about detecting misrepresentations on Web sites in general (M = 3.56, on a 5-point scale), confidence about detecting misrepresentations on the Web site that they had selected as exemplifying misrepresentations (M = 3.43), competence in evaluating the validity of information on their group’s topic (M = 3.57), and competence about evaluating information on the Web in general (M = 3.43). It is important to note that these ratings were elicited only to provide general indications of participants’ levels of certainty of their determinations around credibility. Thus, the ratings do not permit analyses of confidence calibration (Liberman and Tversky 1993; Lundeberg et al. 2000) or discrimination (Lundeberg et al 2000; Lundeberg et al. 1994). To determine whether computer science or education students had higher self-ratings, we ran one-way ANOVAs with grouping as the independent variable and self-ratings for detecting misrepresentations on the Web in general and on evaluating the validity of information on the Web. We expected computer science students to have higher ratings,
123
54
M. K. Iding et al.
since we assumed that they work with Web sites more. There were no significant differences between groups on detecting misrepresentations on Web sites in general, or on evaluating the validity of information on the Web in general. Conclusion Findings from Study 1 highlighted contradictions between what some students viewed as credible and others did not. Students appeared to maintain some confusion about Web site authors’ vested interests, an area that should be investigated further, as users need to make determinations about impetus behind presentation of information on Web sites, and possible biases of Web site authors, even if the Web-based information is educational in nature.
Study 2 Study 2 was carried out to further examine computer science students’ determinations of credibility, vested interests, and confidence in own ratings. In this study, students were provided with three Web sites regarding cleanroom procedures. Method Participants Participants were 25 students in graduate-level computer science courses on human computer interaction and information technology. Materials and procedure The authors developed a survey for use in the present study. In the survey, participants were provided with three Web sites on the topic of ‘‘cleanroom’’ procedures for software development. These Web sites were chosen from the students’ selections of credible Web sites and ones containing misrepresentations from Study 1 (some of the same Web sites that had been selected as credible and accurate by some participants in that study had also been described as inaccurate by other participants). The first Web site was geared toward business and contained advertising. The author of the first Web site was a software consulting service specializing in cleanroom software engineering. The cleanroom approach was briefly described, followed by several paragraphs outlining the consultants’ services. The layout of the page was spare, with few graphics, and the customary menu of links down the left side. A university professor developed the second Web site. It used no graphics at all, and consisted almost exclusively of links to the author’s descriptions and examples of cleanroom software development. The author of a widely used software engineering textbook developed the third Web site. It also used no graphics. This site was a mixture of text and links to additional cleanroom software engineering sites. A heading at the top of the site says ‘‘A collection of Web-based and print resources that will help you understand and explore many different software engineering topics.’’ A list of cleanroom books, and a link to Amazon.com, is at the bottom of the page.
123
Web site credibility
55
In the worksheet, cleanroom engineering is briefly defined: ‘‘The cleanroom technique supports software engineers designing and developing near-zero-defect software. Using the cleanroom techniques, software engineers certify the quality of software, MTTF (mean time to failure) in particular. MTTF is a significant part of end-users’ overall satisfaction.’’ The instructions explained that respondents were not expected to be experts on cleanroom techniques or Web design, but would be asked to give their opinions on whether the Web sites that followed were deemed to be ‘‘objective and accurate representations of cleanroom software engineering…or mistaken and purposely misleading.’’ They were asked to explain why, then asked to rate their confidence in their categorizations on a scale of 1 (no confidence) to 5 (complete confidence). They were also asked to describe each author’s vested interests or motivation. At the conclusion of the survey, participants were asked to ‘‘Think about Web sites in general and answer the following questions: How do you know when a Web site’s content is wrong? On a scale of 1 (no confidence) to 5 (complete confidence), how confident are you about detecting misrepresentations on Web sites in general? On the same 1–5 scale, how competent are you [in] evaluating the validity of information on the Web in general? On a 1 (no competence) to 5 (complete competence) scale, how competent are you evaluating the validity of information about cleanroom software engineering?’’ Coding All written responses were coded collaboratively according to a content analysis scheme developed by two of the authors, and based in part on the scheme used in Study 1 (see categorization scheme in the tables below). Categories emerged from the data and others were adapted from the work of Fogg et al. (2002), as they applied to the present data. Each separate statement that was determined to represent a complete idea was coded, and frequencies were computed for each category. Subcategories for negative and mixed (both negative and positive) comments were also coded. Results and discussion Were Web sites objective and accurate, mistaken, or purposely misleading? Participants determined whether each Web site was objective and accurate, mistaken, purposely misleading, and other. Table 5 shows the frequencies for accuracy ratings for each Web site. It is interesting to note that the majority of answers for the Web sites overall were in the objective and accurate category. Web site 1 (the most obviously commercial site) had the most distributed answers. It had the lowest number (10 or 40%) of ‘‘objective and accurate’’ answers, while 10 comments indicated that participants thought the Web site was ‘‘purposely misleading’’ and 2 comments considered it ‘‘mistaken.’’ Web site 2 (the educational Web site) had 14 comments (or 56% of comments) placing it as ‘‘objective and Table 5 Study 2: Accuracy ratings for clean room Web sites: frequencies and percentages Web site 1 commercial
Web site 2 educational
Web site 3 mixed
Objective and accurate
10 (40%)
14 (56%)
17 (71%)
Purposely misleading
10 (40%)
1 (4%)
2 (8%)
Mistaken
2 (8%)
5 (20%)
2 (8%)
Other
3 (12%)
5 (20%)
3 (13%)
123
56
M. K. Iding et al.
Table 6 Study 2: Reasons for knowing Web site information is inaccurate Category
Frequency for category selection
Expertise of user
12
Corroboration
11
Information designa
6
Information biasa
4
References
4
Information claritya
4
Information focusa
4
Information accuracya
3
Directory to other links
3
Affiliationsa
2
Name recognition/reputationa
2
Performance on test by usera
2
Design looka
2
Readabilitya
1
Commercial/sales
1
Motive of organizationa
1
Not sufficient links Total a
1 63
Indicates categories adapted from Fogg et al. (2002)
accurate,’’ yet only 1 comment indicated it was ‘‘purposely misleading,’’ while 5 comments deemed it as ‘‘mistaken.’’ Web site 3 (another commercial Web site) had 17 comments in the ‘‘objective and accurate’’ category, although 2 comments suggested it was ‘‘purposely misleading,’’ and 2 indicated it was ‘‘mistaken.’’ For both of the commercial Web sites, but particularly Web site 1, the participants attributed an intention to purposely mislead to the Web site. However, when they rated the educational Web site, the participants were more trusting of the Web site designers’ intent and tended to categorize errors as mistakes. After rating the specific Web sites, participants wrote responses to the general question, ‘‘How do you know when a Web site’s content is wrong?’’ The authors coded the major themes that emerged from this data, which are depicted in Table 6. Notably, the category with the largest number of comments was about whether users believed they had expertise in the content area addressed by the site. The next largest category of comments was about corroboration, or whether information agrees with what the user already knows or with other information that can be found. Finally, participants commented on other issues presented such as poor design, bias, problems with references, non-working links, lack of clarity, inaccuracy and sales pitches. Some participants related strategies that they use for determining Web site accuracy. One explained, ‘‘I frequently check multiple Web sites when searching for information, or rely on trusted sites such as the Encyclopedia Britannica, MIT, SourceForge, NY Times.com, etc.’’ Another said, ‘‘By reading the first three paragraphs and subtitles of Web sites.’’ Others raised issues about accuracy in general. As one said, ‘‘In reality, I don’t [know when a Web site is wrong]. The only way that I know is by trying to figure out the
123
Web site credibility
57
motivation for the site and to cross-check information across a number of sites with varying motivations.’’ Another participant raised a similar point, ‘‘There is really no way of knowing that a Web site’s content is wrong, unless I personally know that there is a mistake or inaccuracy in the content based on my own knowledge or experience. The same is true for any book I may check out of the library or the newspaper I read every day. Generally, people tend to believe everything that they see in print, but as an educated human being, I know that this is not always the case. So, just as I need to watch out for inaccuracies in a book or newspaper, I also need to use the same caution when reading a webpage.’’ Finally, a participant raised an issue worth considering, ‘‘The last site’s extensive references to high quality ‘third party’ sites made it seem very credible. This makes me wonder if we can always be sure of the quality of referenced sites. In addition, how would a typical Web surfer even be sure that the referenced sites weren’t merely sites produced by the authors of the original site, designed to mislead and manipulate?’’ Reasons for accuracy decisions In addition to making determinations of Web site accuracy, participants were asked to provide reasons for their answers. Reasons were coded
Table 7 Study 2: Reasons for Web site accuracy determinations Web site
1 Commercial
2 Educational
3 Mixed
Commercial/sales
12 (10N)
0
4 (4N)
Information accuracya
6 (5N)b
6 (2N)
3 (1N)
Information usefulnessa
6 (5N)
3 (3N)
1
Not sufficient links
4 (4N)
2 (2N)
4 (4N)
Design looka
2 (1N)
3 (2N)
1 (1N)
Expertise of user
2
1
0
Information biasa
4 (3N, 1M)c
6 (3N)
5 (2N, 1M)
Information claritya
4 (1N)
9 (4N, 1M)
2 (1N)
Information designa
3 (1M)
2 (1N)
3 (1N)
Educational
2 (1N)
6
4
Readability
2 (1M)
1
0
Currency of informationa
1 (1N)
1 (1N)
1 (1N)
Directory to other links
0
0
7
References
0
1
8
Motive of organization
0
0
0
Name recognition/reputationa
0
1
2 (1N)
Corroboration
0
0
0
Affiliationsa
0
4
0
Personal testimony
0
4
0
Functionality of site
1
1 (1N)
0
Information focusa
0
0
3 (1N)
Totals
49 (31N, 3M)
51 (19N, 1M)
48 (17N, 1M)
a
Indicates categories adapted from Fogg et al. (2002)
b
Indicates negative comment
c
Indicates negative and positive comment
123
58
M. K. Iding et al.
according to the general scheme described earlier. (See Table 7.) Subcategories of negative and mixed comments were also calculated. One of the most notable findings was the range of discrepant reasons for accuracy decisions given by participants. For example, with respect to Web site 1 (the commercial site), 10 of the accuracy ratings (Table 5) had indicated it was objective and accurate, while 10 of these ratings had rated it as mistaken or purposely misleading. In examining reasons for the ratings (Table 7), those who regarded the Web site as purposely misleading frequently referred to the sales aspect in describing it (12 comments fell into this category, 10 were negative). As one participant stated, ‘‘I believe this site is purposefully misleading because they are selling and basically provide no information about what the cleanroom software development really is. They attempt to provide the visitor with the feeling that this technique is necessary in order to sell themselves.’’ In contrast to the suspicion associated with commercial interests, the next largest categories in terms of kinds of reasons for accuracy ratings were about information accuracy (6 comments in total; 5 were negative) and information usefulness (6 comments; 5 negative). One participant made a positive comment about accuracy: ‘‘I believe this Web site is accurate and objective, because the ideas presented are cogent and unambiguous. I checked the links and the same observations are true for the material introduced there as well.’’ Commenting about the accuracy of the same site, another respondent stated, ‘‘It doesn’t tell me what percent of defect—they say ‘nearly defect free—’ (Not accurately conveying info[rmation]).’’ These contrasting perspectives on accuracy could reflect differing levels of familiarity with software engineering procedures, or more simply, different perspectives on the same information. Respondents also made comments about information usefulness, such as, ‘‘The web site didn’t give any details to describe how cleanroom software engineering works and what are the advantages compared to other software engineering methods no[r] any example to show how successful[ly] to implement this technique.’’ Other reasons for determinations fell into categories including insufficient links (4 negative), bias (4 comments; 3 negative; 1 mixed), clarity (4 comments; 1 negative), and information design (3 comments; 1 mixed). This site had the highest number of comments that could be categorized as negative (31). With respect to Web site 2, the educational site, 14 (56%) of accuracy ratings (Table 5) had associated it with the ‘‘objective and accurate’’ category. Most of those who selected this category associated their reasons (Table 7) with the educational nature of the site (6 comments) and 6 of other comments had to do with accuracy (2 were negative). As one respondent cautiously stated, ‘‘I don’t really know how accurate it is, but I would believe that it is for the most part accurate and legitimate since he is coming from an academic background and not selling anything. It certainly does not seem to be purposely misleading.’’ Several commented on personal testimony present in the site (4 comments). These comments were all positive. As one respondent explained, ‘‘From what I read he seems to know what he is talking about. One particular quote helped to convince me that he is happy using the CleanRoom product and appears to be an honest person. The quote is ‘For the first time in my long career, I can honestly say that I have some confidence that my code will run correctly in most cases.’’’ A high number of comments were about information clarity. Four were positive, 4 were negative, and 1 was mixed. Positive comments mentioned clear explanations, ‘‘cogent and unambiguous’’ ideas; whereas negative comments mentioned aspects of user confusion generated by specific kinds of information left out of the site.
123
Web site credibility
59
Finally, although Web site 3 (educational combined with commercial) had the highest number of accuracy ratings (Table 5) describing it as objective and accurate (17 or 71%), participants were mixed in their reasons as well. As one stated, ‘‘The site links to outside information extensively, and while much of it appears to be related, I was uncomfortable with the lack of original content. The link to IBM seemed completely unrelated to classroom techniques, and made me very nervous about the site overall. Also, broken links to bibliographic information contributed to this feeling.’’ In contrast, another said, ‘‘I believe that this Web site is objective and accurate. It is very informative and contains a lot of details, references, suggested readings and links to other related sites. This site clearly explains what the Cleanroom Software Engineering approach is, the history, methods, tutorials and even more. Explanations are scientific, logical and abundant. I feel that this site is honest and trustworthy.’’ Although this site had more objective and accurate ratings than the other sites, it had 17 negative comments associated with it, which was close to the 19 negative comments associated with Web site 2. Web site authors’ vested interests For each of the three Web sites, participants were asked to describe the authors’ ‘‘vested interests.’’ Their comments were coded and the frequencies are shown in Table 8. As expected, the primary category for Web site 1, the commercial site, was commercial/sales (23 comments or 74%), and for Web site 2, the educational Web site, was educational (21 comments or 72%). Finally, Web site 3, the mixed site, was described by 13 comments (45%) as educational, 10 comments (34%) as commercial/sales, and 4 comments to serve as a directory to other links. Although participants were clearly aware of the commercial vested interests associated with Web site 1, and educational vested interests associated with Web site 2, they appeared divided about
Table 8 Study 2: Categories of comments regarding Web site authors’ vested interests Category
Web site 1 commercial Frequency (percentage)b
Commercial/sales
23 (74%)
0
10 (34%)
4 (13%)
21 (72%)
13 (45%) 1 (3%)
Educational Motive of organizationa
Web site 2 educational
Web site 3 mixed
0
1 (3%)
Affiliations
1 (3%)
0
0
Functionality of sitea
1 (3%)
0
0
Information designa
1 (3%)
0
0
0
0
0
Name recognition/reputationa Not sufficient links
1 (3%)
0
0
Personal testimony
0
2 (7%)
0
Information usefulnessa
0
3 (10%)
0
Information accuracya
0
2 (7%)
0
Directory to other links
0
0
4 (14%)
Design looka
0
0
1 (3%)
Information biasa
0
0
0
31
29
29
Total
* Indicates categories adapted from Fogg et al. (2002) ** Indicates frequency per category and percentage of total comments per Web site
123
60
M. K. Iding et al.
the nature of vested interests for Web site 3. This is particularly intriguing, because in this study, as in Study 1, commercial vested interests were regarded with more suspicion than educational interests, which seemed to be frequently perceived as bias-free. Therefore, those who were unaware of commercial vested interests in Web site 3 may have associated it with ‘‘bias free’’ determinations that participants attached to educational Web sites in general. Cleanroom competence and confidence ratings As part of the study, participants rated their confidence in determinations of Web site accuracy for each of the three sites. Specifically, they answered the question, ‘‘On a scale of 1 (no confidence) to 5 (complete confidence), how confident are you about your categorization and answer above?’’ Mean ratings were similar for the Web site 1 (commercial site) (M = 3.76) and Web site 2 (educational) (M = 3.74), and were higher for the third Web site (M = 4.25). Since this site received the highest number of ‘‘objective and accurate’’ ratings, it may be that the association of the site with an academic textbook author on the topic could have been associated with increased confidence. However, the number of participants in this study was limited, providing similarly limited indications of differences in confidence ratings. Participants rated their ‘‘confidence in detecting misrepresentations on Web sites in general’’ as reasonably high (M = 3.60). Next they rated their competence in detecting the validity of information on the Web in general as high as well (M = 3.71). Finally, they rated their competence in ‘‘evaluating the validity of information about cleanroom software engineering’’ as lower (M = 3.06). These findings are interpreted cautiously, as they are not tied to actual accuracy ratings, and have the same limitations that are described with reference to confidence ratings in Study 1. However, self-determinations of confidence and/or competence may play an interesting role in Web site credibility determinations. In particular, university students and others may rely more heavily on self-confidence and competence determinations in credibility judgments about Web resources, especially in areas where they are just becoming acclimated into a discipline or attaining competence (initial stages in developing expertise, according to Alexander’s 2003 model). This would be an interesting area for further research. Directions for further research Areas for further research could also involve examining credibility determinations in conjunction with levels of expertise in different disciplines and in connection with different tasks. For example, how do people’s strategies differ in potentially high cost or high stakes decision-making and in lower cost/stakes scenarios? Other interesting directions for further research could involve disentangling some of the seeming contradictions involved in participants’ different ratings of the same sites by asking more detailed questions, for example, about links to other sites or references. Although it is possible that participants judged the same links differently, it is also possible that they visited different links. Highly controlled laboratory studies (outside the scope of the present research) with sample Web sites and links that are visited by all participants could provide further answers to this question. Cross-cultural comparisons could add interesting information in elucidating whether bases for credibility determinations are the same or different, depending upon cultural context. Further, do these determinations vary across content areas or disciplines?
123
Web site credibility
61
Additionally, examining credibility determinations developmentally is imperative. As Metzger and Flanagin (2008) contend, ‘‘Although research on credibility and new media is burgeoning, extremely little of it focuses on youth (with the exception of college students), in spite of this population’s exceptional immersion in digital technologies’’ (p. 1). Finally, people’s determinations of Web site authors’ vested interests should be studied further, as it appears to be an area about which participants know little. For example some of these participants appeared to presume that educational vested interests are neutral and bias-free. It would be interesting to explore more controversial or political Web sites than the rather neutral cleanroom software engineering sites.
Instructional implications Instructional implications of this work are clearly apparent. These include teaching critical Web site and general information evaluation skills at all levels, practicing with actual Web sites and actual content, and applying strategies to a range of different kinds of content in different areas. Rather than performing brief short-term interventions, Web site evaluation should be addressed repeatedly at appropriate times throughout a school term (Iding et al. 2002b). As Nguyen (2002) has pointed out, many k-12 teachers mistakenly assume that critical Web evaluation skills have already been taught, usually by language arts teachers. Educators may also assume that directly addressing issues of Web credibility is therefore unnecessary. Our present research indicates that even at the university level, improved awareness of factors contributing to Web site and information credibility is needed. In addition to these suggestions, many of which merely reiterate what is known about effective strategy training generally, some specific recommendations emerge form the present research: • Students and others need to learn to critically evaluate information from educational sources as critically as they evaluate information from commercial sites. They need to be aware that even educational materials are imbued with biases. Simply because a site has edu or gov attached to a URL, it certainly cannot considered to be free of bias. • Students also need to consider vested interests of Web site authors. Why was the information placed on the Web? What information is being emphasized and what is being left out (deliberately or otherwise)? How do presentation aspects like layout and design look affect what is being emphasized or de-emphasized? Although commercial vested interests appear to be the easiest to detect, other kinds of influence (e.g., political, etc.) should be considered. • Students need to learn effective information corroboration skills—finding ways to determine information accuracy by comparing it to other information sources. The task is more difficult in areas where we do not possess expertise, but general critical information evaluation skills will be helpful in these areas of non-expertise, as we have seen in the present research. • Students need to check references and links. • Students need to learn about how research is refereed in academic fields and why nonrefereed information may be suspect. • Students and general Internet users need to take on roles as arbiters of information accuracy, rather than merely as seekers of relevant information, despite the fact that making effective accuracy or credibility determinations may take more time and effort.
123
62
M. K. Iding et al.
Conclusion In conclusion, this work contributes to the research on Web site credibility by elucidating reasons university students provide for determining that Web sites are credible or not credible, by studying associated self-ratings of confidence in those determinations and by examining participants’ understandings of vested interests of Web site authors. This general area of research in credibility judgments and Web site evaluation is important as it adds to our understanding of how people determine information on the Web is truthful or not, and provides a basis from which educators can begin to develop ways to address students’ needs for accurate and useful Web evaluation (as well as general information evaluation) skills.
References Alexander, P. A. (2003). The development of expertise: The journey from acclimation to proficiency. Educational Researcher, 32(8), 10–14. Benotsch, E. G., Kalichman, S., & Weinhardt, L. S. (2004). HIV–AIDS patients’ evaluation of health information on the Internet: The digital divide and vulnerability to fraudulent claims. Journal of Consulting and Clinical Psychology, 72(6), 1004–1011. Crosby, M. E., Iding, M. K., Auernheimer, B., & Klemm, E. B. (2002). Judging the veracity of Web sites. Proceedings of the International Conference on Computers in Education (ICCE 2002) (pp. 251–252). Escoffery, C., Miner, K., Adame, D., Butler, S., McCormick, L., & Mendell, E. (2005). Internet use for health information among college students. Journal of American College Health, 53(4), 183–188. Farah, B. (1995). Information literacy: Retooling evaluation skills in the electronic information environment. Journal of Educational Technology Systems, 24(2), 127. Flanagin, A. J., & Metzger, M. J. (2007). The role of site features, user attributes, and information verification behaviours on the perceived credibility of web-based information. New Media Society, 9, 319–342. Fogg, B. J., Soohoo, C., Danielson, D., Marable, L., Stanford, J., & Tauber, E. R. (2002). How do people evaluate a Web site’s credibility: Results from a large study. Retrieved 2 Oct 2008 from http://www. consumerwebwatch.org/dynamic/web-credibility-reports-evaluate-abstract.cfm. Iding, M., & Klemm, E. B. (2005). Pre-service teachers critically evaluate scientific information on the World Wide Web: What makes information believable? Computers in the Schools, 21(1/2), 7–18. Iding, M. K., Auernheimer, B., Crosby, M. E., & Klemm, E. B. (2002a). Users’ confidence levels and strategies for determining Web site veracity. Proceedings of the WWW 2002: The Eleventh International World Wide Web Conference. [CD ROM-Author index, 1–3]. Iding, M., Landsman, R., & Nguyen, T. (2002b). Critical evaluation of scientific websites by high school students. In D. Watson & J. Anderson (Eds.), Networking the learner: Computers in education: Seventh IFIP World Conference on Computers in Education Conference Proceedings. Dordrecht, Netherlands: Kluwer Academic Publishers. Kalichman, S. C., Cherry, C., Cain, D., Pope, H., Kalichman, M., Eaton, L., et al. (2006). Internet-based health information consumer skills intervention for people living with HIV/AIDS. Journal of Consulting and Clinical Psychology, 74(3), 545–554. Klemm, E. B., Iding, M., & Speitel, T. (2001). Do scientists and teachers agree on the credibility of media information sources? International Journal of Instructional Media, 28(1), 83–91. Liberman, V., & Tversky, A. (1993). On the evaluation of probability judgments: Calibration, resolution, and monotonicity. Psychological Bulletin, 114, 162–173. Lundeberg, M.A., Fox, P. W., Brown, A., & Elbedour, S. (2000). Cultural influences on confidence: Country, and gender. Journal of Educational Psychology, 92, 152–159. Lundeberg, M., Fox, P. W., & Puncochar, J. (1994). Highly confident, but wrong: Gender differences and similarities in confidence judgments. Journal of Educational Psychology, 86, 114–121. Metzger, M. J., & Flanagin, A. J. (2008). Introduction. In M. J. Metzger & A. J. Flanagin (Eds.), Digital media, youth, and credibility (pp. 1–4). Cambridge: MIT Press. Metzger, M., Flanagan, A., & Zwarun, L. (2003). College student Web use, perceptions of information credibility, and verification behavior. Computers and Education, 41, 271–290.
123
Web site credibility
63
Nguyen, T. T. (2000). OASIS: Student evaluation methods for World Wide Web resources. Unpublished master’s thesis, University of Hawai‘i, Honolulu, Hawaii, USA. Princeton Survey Research Associates (2002). A matter of trust: What users want from web sites. Retrieved 2 Oct 2008 from http://www.consumerwebwatch.org/dynamic/web-credibility-reports-a-matter-oftrust-abstract.cfm. Rader, H. (1998). Library instruction and information literacy. Reference Service Review, 26(3/4), 143. Stanford, J., Tauber, E. R., Fogg, B. J., & Marable, L. (2002). Experts vs. online consumers: A comparative credibility study of health and finance web sites. Retrieved 2 Oct 2008 from http://www.consumer webwatch.org/dynamic/web-credibility-reports-experts-vs-online-abstract.cfm. Tormala, Z. L., & Petty, R. E. (2004). Source credibility and attitude certainty: A metacognitive analysis of resistance to persuasion. Journal of Consumer Psychology, 14(4), 427–442.
123
