AAH GRAPHICS, INC. / 540-933-6210 / FAX 540-933-6523 / 11-27-2005 / 12:31

TITLE

A Technique for Automatically Scoring Open-Ended Concept Maps AUTHOR

Ellen M. Taricani and Roy B. Clariana ABSTRACT:

In this descriptive investigation, we seek to confirm and extend a technique for automatically scoring concept maps. Sixty unscored concept maps from a published dissertation were scored using a computer-based technique adapted from Schvaneveldt (1990) and colleague’s Pathfinder network approach. The scores were based on link lines drawn between terms and on the geometric distances between terms. These concept map scores were compared to terminology and comprehension posttest scores. Concept map scores derived from link data were more related to terminology whereas concept map scores derived from distance data were more related to comprehension. A step-by-step description of the scoring technique is presented and the next steps in the development process are discussed. Keywords: computer-based assessment, Pathfinder networks, technology tools Currently there is considerable interest in the use of concept maps both to promote and to measure meaningful learning (Shavelson, Lang, & Lewin, 1994). Concept maps are sketches or diagrams that show the relationships among a set of terms by the positions of the terms and by labeled lines and arrows connecting some of the terms. Guided by Ausubel’s (1968) theory, teachers and researchers, mainly in science education, have considered concept hierarchy, proposition correctness, and cross-concept links as the most salient features of concept maps (Rye & Rubba, 2002). Concept maps are most often scored by raters using rubrics to quantify content, meaning, and visual arrangement. Extensive empirical research has shown that scoring approaches with the highest reliability and criterionrelated validity compare specific features in student concept maps to those in expert referent maps (Ruiz-Primo & Shavelson, 1996). Because of the nature of the information in concept maps and also the idiosyncrasies of individual ETR&D, Vol. 54, No. 1, p. 65–82, 2006 © 2006 Association for Educational Communications and Technology.

65

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

66

TARICANI AND CLARIANA

maps, raters almost necessarily must be subject-matter experts in the content of the map. Also, scoring concept maps by hand takes time and experience. McClure, Sonak, and Suen (1999) reported that raters can be overwhelmed by complex scoring rubrics. After comparing six concept map scoring approaches, they concluded that the reliability of concept map scores decreases substantially as the cognitive complexity of the scoring task increases. Taken together, these issues mitigate against the casual use of concept maps for assessment in the classroom with all but the simplest scoring rubrics. Several large-scale projects are underway that seek to automate concept map scoring (Cañas et al., 2003; Herl, O’Neil, Chunga, & Schacter, 1999; Luckie, 2001). In this investigation, we consider a concept map scoring technique that does not require raters and that is based on Pathfinder associative networks (Schvaneveldt, Dearholt, & Durso, 1988). Pathfinder Associative Networks

Pathfinder networks (PFNets) are a well established method for representing knowledge that has been applied in a number of domains of interest to instructional designers (Jonassen, Beissner, & Yacci, 1993). PFNets are two-dimensional graphic network representations of a matrix of relationship data in which concepts are represented as nodes and relationships as unlabeled links connecting the nodes. PFNets visually resemble concept maps, but without linking terms. There are three steps in the Pathfinder approach. (a) In Step 1, raw proximity data are collected typically using a word-relatedness judgment task. Participants are shown a set of terms two at a time, and judge the relatedness of each pair of terms on a scale from 1 (low) to 9 (high). The number of pair-wise comparisons that participants must make is (n2 – n) /2, with n equal to the total number of terms in the list. (b) In Step 2, a software tool called Knowledge Network and Orientation Tool for the Personal Computer (KNOT, 1998) is used to reduce the raw proximity data into a PFNet representation. Pathfinder uses an algorithm to determine a least-weighted path that links all of the terms. The rules for calculating the least-weighted path can be adapted by adjusting parameters that reduce or prune the number of links in the resulting PFNet (refer to Dearholt & Schvaneveldt, 1990). The resulting PFNet is based on a data reduction approach that is purported to represent the most salient relationships in the raw proximity data. (c) In Step 3, the similarity of the participant’s PFNet to an expert referent PFNet is calculated also using KNOT software (Goldsmith & Davenport, 1990). The total number of derived links

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

SCORING OPEN-ENDED CONCEPT MAPS

67

shared by two PFNets is called links in common. Common is a positive integer that ranges from 0 to the maximum number of links in the referent PFNet. How can this Pathfinder approach be used to score concept maps? Several investigators have shown that concept-map–like tasks capture some of the same types of relational information as word-relatedness judgment tasks (Jonassen, 1987; McClure et al., 1999; Rye & Rubba, 2002; Schau, Mattern, Zeilik, & Teague, 1999). Compared to word-relatedness judgment tasks, concept maps are relatively fast to complete and perhaps more authentic (Shavelson et al., 1994). In the present investigation, concept maps provide an alternative to word-relatedness judgment tasks in Step 1 for obtaining raw proximity data, whereas Steps 2 and 3 are conducted in the conventional way. Thus, the main contribution of this investigation is in clearly describing how components of a concept map can be converted into raw proximity data in Step 1, and then describing the validity and reliability of the scores obtained. An Automatic Technique for Scoring Concept Maps

What information components of concept maps can be collected automatically? Concepts, links, and linking terms can be counted in various ways. In addition, following Kinchin (2000), Yin, Vanides, Ruiz-Primo, Ayala, and Shavelson (2004) have proposed that map structure complexity, as determined by examining the overall visual layout of the map, should also be considered. Automatically measuring these components of concept maps is easier with closed-ended concept mapping tasks, where the student is provided with a predefined list of concepts and linking terms, such as the concept map scoring software used by Herl et al. (1999). However, many investigators refer to openended concept mapping, where participants may use any concepts and linking terms in their maps, as the gold standard for capturing students’ knowledge structures (McClure et al., 1999; Ruiz-Primo, Schultz, Li, & Shavelson, 1999; Yin et al., p.24). But automatically scoring open-ended concept maps is considerably more difficult. Clariana, Koul, and Salehi (in press) piloted a technique for scoring openended concept maps. Practicing teachers enrolled in graduate courses constructed concept maps on paper while researching the topic, “the structure and function of the human heart and circulatory system,” online. Participants were given the online addresses of five articles that ranged in length from 1,000 to 2,400 words, but were encouraged to view additional resources. After completing their research, participants then used their concept map as an outline to write a 250-word text summary of this topic (see Clariana, 2003). Computer software tools (Clariana, 2002) were used to measure the geometric distances between terms in the concept maps, referred to as distance data, and

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

68

TARICANI AND CLARIANA

to count the link lines that connected terms, referred to as link data (see Figure 1). Using Pathfinder KNOT software, the raw distance and link data were converted into PFNets and then were compared to an expert’s PFNets to obtain similarity scores. Five pairs of raters using rubrics also scored all of the concept maps and text summaries. The correlation values (Pearson’s r) for the concept maps scored by raters compared to concept map link-based scores was 0.36, to concept map distance-based scores was 0.54, and to text summaries scored by raters was 0.49. The correlation values for the text summaries scored by raters compared to concept map link-based scores was 0.76 and to concept map distance-based scores was 0.71. The authors concluded that these “automatically derived concept map scores can provide a relatively low-cost, easy to use, and easy to interpret measure of students’ science content knowledge.” It is important to note that link line does not correspond to the term proposition used in the concept map literature. A proposition is the combination of two concepts (e.g., subject-predicate) and a linking term (e.g., verb) that describes the relationship between the two concepts. For example, in the proposition, “the aorta is a blood vessel,” aorta and blood vessel are concepts and is a is the linking term. Typically, propositions are scored based on correctness, which includes deciding whether the linking term is valid and significant for that context. Recently, Harper, Hoeft, Evans, and Jentsch (2004) reported that the correlation between just counting link lines and actually scoring correct and valid propositions in the same set of maps was r = 0.97, suggesting that the substantial extra time and effort required to specify and hand-score all possible linking terms adds little additional information over just counting link lines.

Figure 1

The link and distance data arrays of a simple map.

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

SCORING OPEN-ENDED CONCEPT MAPS

69

Thus, the approach used by Clariana et al. (in press) is a significant departure from most concept map scoring approaches because it considers link lines, not proposition correctness. In addition, their technique uses the relative spatial location of concepts to communicate hierarchical and coordinate concept relations (Robinson, Corliss, Bush, Bera, & Tomberlin, 2003, p.26). This approach is founded on previous research on free association norms (Deese, 1965), on structural analyses of text propositions (Frase, 1969), on the matrix model of memory (Humphreys, Bain, & Pike, 1989; Pike, 1984), and on current neural network models of cognition (McClelland, McNaughton, & O’Reilly, 1995). For example, in an untrained neural network, concept units (e.g., subjects, predicates, and linking words) are randomly associated, but as the network learns by experiencing many propositions, structural relationships emerge. Elman (in press) has shown that verbs (linking words) and nouns (subjectspredicates) separate into two separate clusters, and then nouns subcluster based on their meaning (e.g., animate and inanimate). In the present investigation, we assumed that the distances between concepts terms in concept maps captures aspects of this underlying fundamental concept structure (e.g., subject-predicate associations). In addition, distance data may provide a direct measure of what Yin et al. (2004) called “map structure complexity.” In a follow-up study, Clariana and Poindexter (2004) used the same Pathfinder scoring technique but asked participants to draw network maps rather than concept maps. Network maps are like concept maps, except that there are no linking terms. The mapping directions specifically directed the participants to use spatial closeness to show relationships and intentionally deemphasized the use of link lines. Participants completed one of three print-based text lessons on the heart and circulatory system. The three lesson treatments included adjunct constructed response questions, scrambled sentences, and a readingonly control. Participants completed three multiple-choice posttests that assessed identification, terminology, and comprehension and then were handed a list of 25 preselected terms and were asked to draw a network map. The adjunct question treatment was significantly more effective than the other lesson treatments for the comprehension outcome, and no other treatment comparisons were significant. Scores based on network map link data (relative to distance data) were more related to terminology, with Pearson’s r = .77 compared to r = .69, while scores based on network map distance data (relative to link data) were more related to comprehension, with Pearson’s r = .71 compared to r = .53. Thus the geometric distances between terms related more to the broader processes and functions of the heart and circulatory system, while the links drawn to connect terms related more to verbatim knowledge from the lesson text covering facts, terminology, and definitions.

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

70

TARICANI AND CLARIANA

The purpose of the present descriptive investigation is to confirm and extend these two previous experimental studies involving this computerbased technique for scoring open-ended concept maps. The ultimate goal of this line of research is to develop a software tool that allows learners to create and save a concept map and then automatically score the map relative to an archived expert referent map, receiving specific feedback on their maps. One necessary component of this software tool is the mathematical approach used to convert raw data into scores in Step 2. Therefore, in this investigation we compare multiple data reduction approaches for forming PFNets. The resulting scores are compared to traditional multiple-choice posttests that measure terminology and comprehension of the lesson content as a measure of the concurrent criterion-related validity of the various concept map scores.

METHOD

A recent dissertation that used concept maps as an instructional treatment provided an ideal and convenient existing data set for the purposes of the present investigation (Taricani, 2002). Though this is previously published data, in the present investigation we do not reexamine the original research questions. Further, this computer-based scoring method was not available at the time of the original publication and the concept maps were not scored by raters in that study. In Taricani’s dissertation, undergraduate students were randomly assigned to one of five treatment conditions including (a) a learnergenerated concept map treatment with feedback, (b) a learner-generated concept map treatment without feedback, (c) a partially completed fill-in-the blank concept map treatment with feedback, (d) a partially completed fill-inthe blank concept map treatment without feedback, and finally (e) a readingonly, no map or feedback control treatment. Of these five, only the first two treatments involved creating a concept map, so only these two treatments are included in the present investigation. Participants

Participants were freshmen students at a large northeastern university (n = 60) recruited as volunteers from both science and nonscience courses. They were randomly assigned to either the learner-generated concept map treatment with feedback (Feedback) or the learner-generated concept map treatment without feedback (No Feedback). The No Feedback treatment group had 17 males and 13 females and the Feedback group, 12 males and 18 females. Participation was voluntary, and participants were rewarded with either extra

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

SCORING OPEN-ENDED CONCEPT MAPS

71

course credit or pizza and ice cream for their participation but not for performance. Materials

The print-based instructional text on the human heart was a 1,900-word passage called “The human heart: Parts of the heart, circulation of blood, and cycle of blood pressure,” developed by Dwyer (1972). The text dealt with the parts of the heart and the internal processes that occur during the systolic and diastolic phases. The complexity of the information provided was suitable for these participants’ general knowledge and comprehension levels. Following the approach used by Novak and Gowin (1984), a two-page lesson on how to create a concept map was developed (available from Taricani, 2002, pp. 122–124). This two-page lesson included a description of concept mapping and an example of a hierarchical concept map. A short paragraph was presented that contained a familiar scenario about a student who took a walk to the campus library. The walk scenario was selected as a metaphor of blood flow through the heart (e.g., the student moves from point to point, going past various buildings along the way, and blood flows from point to point, passing through various components along the way). After reading this paragraph, participants were asked to draw a concept map of the library walk scenario on a blank sheet of paper as practice. To foreshadow the lesson treatment, feedback in the form of an instructor-prepared hierarchical concept map of the library walk was given to the Feedback group after they had completed the drawing portion of the two-page concept map lesson. Feedback was not provided to the No Feedback group. Posttests

The multiple-choice criterion posttest originally developed and validated by Dwyer (1972), consisted of 20 questions that dealt with terminology and 20 questions that dealt with comprehension. The terminology test was designed to measure declarative knowledge of facts, terms, and definitions. The comprehension test was designed to measure a more thorough understanding of the processes of the human heart, with a specific focus on the functions of different parts of the heart. The KR-20 reliability for the posttest was 0.83. Procedure

The procedure was similar for both the Feedback and No Feedback treatment groups. First, participants completed a demographic survey. Next they completed the two-page training lesson on how to draw a concept map, with or

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

72

TARICANI AND CLARIANA

without feedback. Then participants read the three-page instructional text on the human heart and were asked to draw a concept map of that information on a blank piece of paper while reading the lesson text. Participants could use any terms and any number of terms in their maps. After completing their concept maps, the Feedback group was given an instructor-generated hierarchical concept map as feedback. They were asked to compare this feedback map to their own map and were allowed to change their map using pens of a different color. Only three of the participants made changes to their concept maps after viewing the feedback map. Finally, the lesson materials and concept maps were collected and then all participants completed the multiple-choice posttests. Establishing the Expert Concept Map Scoring Referent

An optimized expert referent concept map is required for comparison during the scoring process (i.e., in Step 3). With open-ended concept maps, participants may use any terms and any number of terms in their maps, but only those terms also included in an expert referent (including synonyms and metonyms) can count during scoring. In order to optimize the value of the information across the entire set of 60 concept maps, a frequency list containing all of the terms used by the participants was prepared. A common term, such as left ventricle, had a high frequency count, while a term such as white corpuscles had a low frequency count. Functionally equivalent terms, such as auricle and atrium, were listed together as acceptable synonyms. A biology content expert was provided with the instructional text and the frequency list of terms and was asked to draw a concept map of the structure and function of the human heart and circulatory system. The expert was not required to use terms from the list, though the list was intended to influence the expert’s choice of terms. The resulting optimized expert referent concept map contained 26 terms with 36 links between terms. Recall that the instructor-generated concept map that served as feedback during the lesson was hierarchical, following the requirements of Novak and Gowin (1984). However, the optimized expert referent concept map developed after the fact for scoring purposes was clearly not hierarchical; it had a network structure that consisted of a cluster of terms in the center, and two flow loops, one to the left and the other to the right of the central cluster. This nonhierarchical network structure for the expert’s map is supported by RuizPrimo and Shavelson (1996) and Safayeni, Derbentseva, and Cañas (2003), who have questioned the requirement of imposing hierarchical structure on concept maps.

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

SCORING OPEN-ENDED CONCEPT MAPS

73

Step 1—Collecting Raw Concept Map Proximity Data Link Data

Concept map link data were collected following Clariana et al. (in press). First, the links on each concept map were entered into an array that can be analyzed by KNOT software. Each array listed the 26 terms used by the expert down the left column and across the top row, so that every cell in the array represents the relationship between 2 of the 26 terms. Following convention, since the data above the diagonal are redundant with data below the diagonal, each link array consisted of just the 325 elements in the lower triangle of the array (e.g., (262 – 26)/2). A 1 entered in a link array cell indicates that a link was drawn between those two terms on the concept map while a 0 indicates that no link was drawn. Referring back to Figure 1, because a link is drawn between left atrium and pulmonary vein, then a 1 is entered into the corresponding cell in the link array. Linking terms were not considered, such as is a, has a and contains; only the presence or absence of each link line (ink on paper) was recorded. Link raw data are similarity data, where smaller values indicate weaker relationship. When a student did not include 1 of the 26 terms, all possible links to that term were coded as 0 to indicate no relationship to the other terms. The interrater reliability for transferring links from the paper-based maps to the link array was 0.98. Distance Data

Concept map distance data were collected using ALA-Mapper software (Clariana, 2002). ALA-Mapper converts the spatial location of the terms in a concept map into a distance array containing all of the pair-wise distances between terms (refer back to Figure 1). Each paper-based concept map was recreated in ALA-Mapper, a manual process that can introduce error, though care was taken to maintain the original spatial proportional relationships between the terms in the map. The interrater reliability for this process was 0.81. Distance data are dissimilarity data, where larger values indicate weaker relationship. When a student did not include 1 of the 26 terms, all possible links to that term were coded as ∞ to indicate no relationship to the other terms. Step 2—Reducing Raw Concept Map Data into PFNets

KNOT software was used to establish link and distance-based PFNets for each concept map by selecting the command ANALYZE PROXIMITY DATA found in the Data main menu. To establish PFNets, KNOT software requires information that describes the raw data and also two calculation parameters. The required

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

74

TARICANI AND CLARIANA

descriptive information includes (a) the data structure (e.g., lower triangle), (b) whether data are dissimilarity or similarity data (e.g., the link data are similarity data and the distance data are dissimilarity data), (c) the number of decimal places in the data (e.g., 0 for both data sets), (d) the maximum value (e.g., link data set to 1 and distance data set to 1,000), (e) the minimum value (e.g., 0.1 for both data sets). Note that the maximum of 1,000 for distance data and the minimum of 0.1 for link data are necessary in order to exclude missing terms from the analysis. In addition, two KNOT software parameters, q and Minkowski’s r, are used to define the rules for including and removing links, thus these two parameters are used to set the amount of data reduction that is desired. Minkowski’s r can range from 1 to ∞ and q can range from 2 to n – 1 (where n is the number of terms). The r parameter is used in the calculation of the weight between terms (Dearholt & Schvaneveldt, 1990, p. 3). The q parameter is used to set the depth of the path that is considered during link pruning. Changing the r and q parameters usually results in changes in the resulting PFNets. Using the largest meaningful values of r and q produces a PFNet with the fewest links (the most data reduction) while using the smallest values of r and q produces a PFNet with the most links (the least data reduction). The link raw data in this investigation provides an interesting exception. Because the link raw data consists only of 1s and 0s, identical PFNets will be produced no matter what values are set for the r and q parameters. Further, there is no data reduction when creating these link-based PFNets. Here, the expert’s link raw data contains 36 links and the expert’s link-based PFNet contains the same 36 links. For link data, the r and q parameters were set to the maximum values, with r equal to ∞ and q equal to 25 (i.e., n – 1) because these are the recommended parameter values for ordinal level data. What values of the r and q parameters are appropriate for raw concept map distance data? Few specific guidelines exist for which values of Minkowski’s r are best for what situation (Durso & Coggins, 1990). Most investigations use the maximum values of r (∞) and q (n – 1), designated in the form PFNet(r, q), perhaps because this should reveal the most salient information in the raw data and also because most research involving Pathfinder uses ordinal-level data. Rather than use one set of values, Dearholt and Schvaneveldt (1990) stated, “Frequently, important information from a given set of proximity data can be obtained from different PFNets, constructed using different values of r and q” (p. 5). Following this recommendation, four separate sets of PFNets were established from raw distance data across a range of Minkowski’s r values including 1, 2, 3, and ∞. Regarding the q parameter, because distance raw data are the actual distances between terms in two-dimensional space, the value used for q is not as critical (regarding violations of the triangle inequality

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

SCORING OPEN-ENDED CONCEPT MAPS

75

in Euclidean space; see Buzydlowski, 2002, p. 27). Thus, following convention, the q parameter was set to its maximum meaningful value of 25 (i.e., n – 1). Step 3—Comparing PFNets to the Optimized Expert Referent

KNOT software was used to compare participants’ PFNets to the optimized expert referent PFNet by selecting the command ANALYZE NETWORKS found in the Network main menu. The software provides two highly correlated measures of similarity called common and configural similarity. Common is a global measure of similarity that is simply the total number of links shared by two PFNets. Common is reported in this investigation because it is easier to understand and also because it was a better predictor than configural similarity in the two previous studies. Reliability of the Concept Map Scores

Because the expert’s link-based PFNet contains 36 links, participants’ maximum possible score is 36 (see the column labeled Maximum Possible in Table 1). These link-based scores may be viewed as a 36-item test. A test item response table with 37 columns × 60 rows was created. Column 1 listed the names of the 60 participants while columns 2 through 37 listed the performance of each student for each of the expert’s 36 links. Students received a 1 in the appropriate cell when their link agreed with the expert and a 0 when it did not. For the linkbased scores with 36 possible links, Cronbach’s alpha was 0.86 for the No Feedback group and 0.76 for the Feedback group. In the same way, a test item response table for the distance-based scores was created (34 maximum possible for PFNet(2,25)). Cronbach’s alpha was 0.80 for the No Feedback group and 0.66 for the Feedback group. Thus, the link-based scores are a little more reliable than the distance-based scores, and both link-based and distance-based scores for the No Feedback group are more reliable than those of the Feedback group.

RESULTS AND DISCUSSION

Though not of central interest in this present analysis, the seven posttest and concept map dependent variables listed in Table 1 were analyzed by multivariate analysis of variance (MANOVA) to determine whether the Feedback and No Feedback group means were different. Though the No Feedback group means for both the multiple-choice and the concept map scores tended to be a little larger than those of the Feedback group (see Table 1), none of the seven dependent variables was significantly different. For example, the greatest between-subjects effect was observed for comprehension posttest scores, with

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

76

TARICANI AND CLARIANA

F (1,59) = 2.887, MSe = 0.038, p = .09. The remainder of this section will describe the relative characteristics of the concept map scores. Estimating the Concurrent Criterion-related Validity of the Concept Map Scores

Correlation analysis is a common approach used for estimating concurrent criterion-related validity (Crocker & Algina, 1986). Pearson’s correlation was used to compare the terminology and comprehension multiple-choice criterion posttests to each concept map score shown in Table 2. First, the No Feedback and Feedback groups obtained very different values and patterns of correlations in Table 2 and few of the Feedback group correlations were significant. The low correlations observed for the Feedback group between multiple-choice posttest scores and concept map scores can be at least partially accounted for by attenuation due to the lower test reliability of the Feedback group’s concept map scores, especially for distance-based scores. Further, it is not obvious that concept map scores should strongly correlate with multiple-choice test scores, thus Pearson’s r values in the 0.30 to 0.40

Table 1

Means and standard deviations for the multiple-choice posttests and the alternate concept map scores. Maximum Possible

Observed Range

No Feedback Group M SD

Feedback Group SD M

Terminology

20

1 to 19

9.5

(4.6)

8.3

(4.1)

Comprehension

20

2 to 19

8.7

(4.1)

7.0

(3.7)

PFNet (∞, 25)

36

1 to 27

11.5

(6.4)

11.8

(5.1)

Distance data PFNet (1, 25)

65

1 to 65

23.4 (13.4)

PFNet (2, 25)

34

1 to 21

10.3

(5.2)

9.5

(4.0)

PFNet (3, 25)

31

1 to 17

9.2

(4.5)

8.4

(3.7)

PFNet (∞, 25)

28

1 to 16

8.4

(4.2)

7.7

(3.4)

Multiple-Choice Posttests

Concept Map Scores Link data

Note: PFNet = Pathfinder network.

27.3 (15.0)

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

77

SCORING OPEN-ENDED CONCEPT MAPS

Table 2

Correlation analyses of concept map scores with the multiple-choice posttests. No Feedback

Feedback

Term

Comp

Term

Comp

0.78*

0.54*

0.46*

0.30

PFNet (1, 25)

0.32

0.45*

0.02

0.05

PFNet (2, 25)

0.48*

0.61*

0.26

0.11

PFNet (3, 25)

0.40*

0.53*

0.36

0.18

PFNet (∞, 25)

0.38*

0.49*

0.38*

0.22

Link data PFNet (∞, 25) Distance data

* p < .05; Term—terminology posttest; Comp—comprehension posttest)

range seem reasonable and even expected. Thus establishing both convergent and discriminant validity is an important step in understanding what concept maps measure. However, low correlations for the Feedback group may also be partially accounted for by the negative effect of concept maps as feedback (Tan, 2000). Recall that the main difference between the Feedback and No Feedback group is the instructor-generated hierarchical concept map given as feedback to the Feedback group. Lee and Nelson (2005) suggested that presenting completed concept maps to learners as instructional materials places emphasis on “. . . unintentional imitation of its structure by the learners because the learners are given a concept map and cannot organize the concept map to make it fit with their internal knowledge structures” p. 194. Similarly, Lambiotte and Dansereau (1992) reported that, “students with more well established schemas for the circulatory system performed less well when structure was imposed by an outline or a [concept] map” (p. 198). Safayeni et al. (2003) went so far as to suggest that hierarchical concept maps of the circulatory and other biological systems are incompatible with the actual understood nature of those systems (p. 12) and proposed a new form of concept map called cyclic concept maps (Kinchin, 2000). Further, neither the participants nor the expert created hierarchical concept maps of this content, and so the instructor-generated hierarchical concept map given as feedback was likely difficult to reconcile because it was so different from the students’ nonhierarchical maps. In hindsight, providing this hierarchical concept map as feedback was ill-advised. Future

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

78

TARICANI AND CLARIANA

research should consider the effects of using cyclical versus hierarchical concept maps as lesson feedback. Second, scores derived from concept map link data were, in general, more related to multiple-choice posttest performance than were those derived from concept map distance data. Again, this can be at least partially accounted for by attenuation due to the lower reliability of the distance-based concept map scores. On the other hand, because it takes extra work to draw the link lines, links drawn on paper are probably more salient to the individual creating the concept map than are the distances between terms. Third, identical to the findings reported by Clariana and Poindexter (2004), for the No Feedback group, distance-based scores relative to link-based scores were a better predictor of comprehension: r = .61 compared to r = .54, and linkbased scores relative to distance-based scores were a better predictor of terminology scores: r = .78 compared to r = .48 (see Table 2). This finding suggests that link and distance data in the same map are capturing different aspects of participant knowledge.

GENERAL DISCUSSION

This investigation confirms a technique for scoring open-ended concept maps using a PFNet analysis approach. The concept map score reliability and the criterion-related correlation results indicate that the technique is capturing different aspects of participant knowledge of the lesson content. Most automatic concept map scoring systems focus on proposition correctness, which totally depends on the correctness of the linking term connecting the two concepts (see Cañas et al., 2003; Herl et al., 1999; Luckie, 2001). Based on our connectionist view for representing knowledge, the technique applied in this investigation totally disregards linking terms, yet manages to capture information. In addition, our technique is different from other current automatic scoring approaches in that (a) it represents concept map raw data in vector arrays, which allows for easier manipulation, aggregation, reduction, comparison, and analysis; (b) it provides a measure of map structural complexity that likely reflects the structure of the content; (c) participants’ concept maps can easily be compared to multiple distinct forms of expert concept maps and to each other as required by the researcher; and (d) pragmatically, it can be applied to both closed and open-ended concept maps. The instructional content and the centrality of the expert referent map in this investigation limit the generalizability of these findings. The lesson content involved blood flow through the circulatory system, which is essentially a

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

79

SCORING OPEN-ENDED CONCEPT MAPS

cyclical path, and PFNets are path representations (Safayeni et al., 2003). This concept map scoring technique may not be adequate for content that is not path-like. Also, the quality of the expert referent map is critical to the quality of the analyses because every map is compared to the referent map. Future research should consider how consistently superior referent maps can be established, perhaps by using multiple expert maps. The ultimate goal of this line of research is to develop a software tool that can be used to create, save, and automatically score concept maps, providing feedback on acceptable and less acceptable components of the map. Future research should consider using weighted and directional link lines. Students could select links that are dashed, thin, and thick to indicate degree of relatedness, and single- or double-headed arrows to show the directionality of the link relationship. This type of data can be easily represented in the data arrays used in this investigation, and KNOT software can already accommodate this type of analysis. An important module of this software tool is the mathematical approach used to convert raw concept map data into scores. Besides PFNets, future research should consider alternate scoring approaches for directly analyzing raw link and distance data such as correlation, factor analysis, and multidimensional scaling, as well as approaches for comparing visual representations from the fields of biology (e.g., protein visualization tools) and engineering (e.g., data mining). Further refinement of this technique for scoring concept maps is warranted.

Ellen M. Taricani and Roy B. Clariana [[email protected]] are with the College of Education at The Pennsylvania State University.

REFERENCES Ausubel, D. P. (1968). Educational psychology: a cognitive view. New York, NY: Holt, Rinehart, and Winston. Buzydlowski, J. W. (2002). A comparison of self-organizing maps and Pathfinder networks for the mapping of co-cited authors. Doctoral dissertation completed at Drexel University. Retrieved October 3, 2004, from http://faculty.cis.drexel.edu/jbuzydlo/papers/thesis.pdf. Cañas, A. J., Coffey, J. W., Carnot, M. J., Feltovich, P., Hoffman, R. R., Feltovich, J., & Novak, J. D. (2003). A summary of literature pertaining to the use of concept mapping techniques and technologies for education and performance support. A report for The Chief of Naval Education and Training at Pensacola, Florida. Retrieved October 3, 2004, from http://www.ihmc.us/users/aCañas/Publications/ConceptMapLitReview/IHMC /LiteratureReviewonConceptMapping.pdf. Clariana, R. B. (2002). ALA-Mapper, version 1.0. Retrieved October 3, 2002, from http://www.personal.psu.edu/rbc4/ala.htm.

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

80

TARICANI AND CLARIANA

Clariana, R. B. (2003). Latent semantic analysis writing prompt, used with the permission of Knowledge Analysis Technologies, LLC. Retrieved October 30, 2003 from http://www.personal.psu.edu/rbc4/frame.htm. Clariana, R. B., Koul, R., & Salehi, R. (in press). The criterion related validity of a computer-based approach for scoring concept maps. International Journal of Instructional Media, 33(3), in press. Clariana, R. B., & Poindexter, M. T. (2004). The influence of relational and proposition-specific processing on structural knowledge. Presented at the annual meeting of the American Educational Research Association, April 12–16, San Diego, CA. Retrieved April 15, 2004 from http://www.personal.psu.edu/rbc4/aera_2004.doc. Crocker, L. A., & Algina, J. (1986). Introduction to classical and modern test theory. New York, NY: Holt, Rinehart, and Winston. Dearholt, D. W. & Schvaneveldt, R. W. (1990). Properties of Pathfinder Networks. In Schvaneveldt (Ed.), Pathfinder associative networks: studies in knowledge organization (pp. 1–30). Norwood, NJ: Ablex Publishing Corporation. Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: The Johns Hopkins Press. Durso, F. T., & Coggins, K. A. (1990). Graphs in the social and psychological sciences: empirical contributions of Pathfinder. In Schvaneveldt (Ed.), Pathfinder associative networks: Studies in knowledge organization (pp. 31–51). Norwood, NJ: Ablex Publishing Corporation. Dwyer, F. M. (1972). A guide for improving visualized instruction. State College, PA: Learning Services. Elman, J. L. (in press). An alternative view of the mental lexicon. Trends in Cognitive Science, in press. Retrieved December 10, 2004 from http://www.crl.ucsd.edu/elman/Papers/elman_tics_opinion_2004.pdf. Frase, L. T. (1969). Structural analysis of the knowledge that results from thinking about text. Journal of Educational Psychology, 60(6, Part 2), 1–16. Goldsmith, T. E., & Davenport, D. M. (1990). Assessing structural similarity of graphs. In Schvaneveldt (Ed.), Pathfinder associative networks: studies in knowledge organization (pp. 75–87). Norwood, NJ: Ablex Publishing Corporation. Harper, M. E., Hoeft, R. M., Evans, A. W., & Jentsch, F. G. (2004). Scoring concepts maps: Can a practical method of scoring concept maps be used to assess trainee’s knowledge structures? Paper presented at the Human Factors and Ergonomics Society 48th Annual Meeting. Herl, H. E., O’Neil, H. F. Jr., Chunga, G. K. W. K., & Schacter, J. (1999). Reliability and validity of a computer-based knowledge mapping system to measure content understanding. Computers in Human Behavior, 15, 315–333. Humphreys, M. S., Bain, J. D., & Pike, R. (1989). Different ways to cue a coherent memory system: A theory for episodic, semantic and procedural tasks. Psychological Review, 96, 208–233. Jonassen, D. H. (1987). Assessing cognitive structure: verifying a method using pattern notes. Journal of Research and Development in Education, 20(3), 1–13. Jonassen, D. H., Beissner, K., & Yacci, M. (1993). Structural knowledge: techniques for representing, conveying, and acquiring structural knowledge. Hillsdale, NJ: Lawrence Erlbaum Associates. Kinchin, I. M. (2000). Using concept maps to reveal understanding: A two-tier analysis. School Science Review, 81, 41–46. KNOT. (1998). Knowledge Network and Orientation Tool for the Personal Computer, version 4.3. Retrieved October 3, 2003 from http://interlinkinc.net/.

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

SCORING OPEN-ENDED CONCEPT MAPS

81

Lambiotte, J., & Dansereau, D. (1992). Effects of knowledge maps and prior knowledge on recall of science lecture content. Journal of Experimental Education, 60, 189–201. Lee, Y., & Nelson, D. (2005). Viewing or visualizing—which concept map strategy works best on problem solving performance? British Journal of Educational Technology, 36(2), 193–203. Luckie, D. (2001). C-TOOLS: Concept-Connector Tools for online learning in science. A National Science Foundation grant. Retrieved October 3, 2003 from http://www.msu.edu/luckie/CTOOLS.pdf. McClure, J., Sonak, B., & Suen, H. (1999). Concept map assessment of classroom learning: reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36, 475–492. McClelland, J. L., McNaughton, B. L., & O’Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models. Psychological Review, 102, 419–457. Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. Cambridge, UK: Cambridge University Press. Pike, R. (1984). A comparison of convolution and matrix distributed memory systems. Psychological Review, 91, 281–294. Robinson, D. H., Corliss, S. B., Bush, A. M., Bera, S. J., & Tomberlin, T. (2003). Optimal presentation of graphic organizers and text: A case for large bites? Educational Technology Research and Development, 51(4), 25–41. Ruiz-Primo, M. A., & Shavelson, R. J. (1996). Problems and issues in the use of concept maps in science assessment. Journal of Research in Science Teaching, 33, 569–600. Ruiz-Primo, M. A., Schultz, S., Li, M., & Shavelson, R. J. (1999). On the cognitive validity of interpretations of scores from alternative concept mapping techniques. (CSE Tech. Rep. No. 503). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Rye, J. A., & Rubba, P. A. (2002). Scoring concept maps: an expert map-based scheme weighted for relationships. School Science and Mathematics, 102, 33–44. Safayeni, F., Derbentseva, N., & Cañas, A. J. (2003). Concept maps: a theoretical note on concepts and the need for cyclic concept maps. Retrieved October 3, 2004 from http://cmap.ihmc.us/Publications/ResearchPapers/Cyclic%20Concept%20Maps. pdf. Schau, C., Mattern, N., Zeilik, M., & Teague, K. W. (1999). Select-and-fill-in concept map scores as a measure of students’ connected understanding of introductory astronomy. Paper presented at the Annual Meeting of the American Educational Research Association (Montreal, Canada, April). Available from ERIC ED434909. Shavelson, R. J., Lang, H., & Lewin, B. (1994). On concept maps as potential authentic assessment is science. (CSE Tech. Rep. No. 388). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge organization. Norwood, NJ: Ablex Publishing Corporation. Schvaneveldt, R. W., Dearholt, D. W., & Durso, F. T. (1988). Graph theoretic foundations of Pathfinder networks. Computers and Mathematics with Applications, 15, 337– 345. Tan, S. C. (2000). The effects of incorporating concept mapping into computer assisted instruction. Journal of Educational Computing Research, 23, 113–131.

AAH GRAPHICS, INC. / (540) 933-6210 / FAX 933-6523 / 10-24-2005 / 18:49

82

TARICANI AND CLARIANA

Taricani, E. M. (2002). Effects of the level of generativity in concept mapping with knowledge of correct response feedback on learning. Dissertation Abstracts International, ISBN # 0-49367101-3. Yin, Y., Vanides, J., Ruiz-Primo, M. A., Ayala, C. C., & Shavelson, R. J. (2004). A comparison of two construct-a-concept-map science assessments: created linking phrases and selected linking phrases. (CSE Tech. Rep. No. 624). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).