Snap Shots
Do I Know What I Do Not Know? Self-Evaluation of Performance in Student-Run Seminars by Psychiatry Trainees in India
Received November 2, 2011; revised February 20, June 1, July 22, 2012; accepted August 9, 2012. From the Dept. of Psychiatry, Dept. of Biostatistics, National Institute of Mental Health and Neurosciences (NIMHANS), Bangalore, Karnataka, India. Send correspondence to Prabha Chandra, M.D.; e-mail:
[email protected] Copyright © 2012 Academic Psychiatry
seminar was chosen as a medium for examining selfevaluation, because it is a pragmatic and commonly-used method of training in most psychiatry training centers in India. The study was conducted at the National Institute of Mental Health and Neurosciences at Bangalore, in India, a center for postgraduate psychiatry training. Trainee performance was assessed in 50 resident-led seminar presentations, using a 12-item, semi-structured instrument developed for the purpose. The tool was developed by consensus among the faculty members in charge of training, after discussions and a literature search regarding presentation skills and assessment. This reduced the risk of individual bias and enhanced content validity. Internal consistency (a necessary condition for construct validity) of this tool was acceptable. A Cronbach’s alpha of 0.947 demonstrates that the 12 items in the assessment rubric consistently measured students’ presentation skills. A high degree of internal consistency demonstrated by the assessment rubric also indicates good convergent validity. The instrument consists of 12 items, assessing completeness (breadth of coverage of topic), relevance of content to topic, organization of slides, time-management, presentation style, creativity in presentation (use of flow-charts, tables, etc.), handling questions from the audience, summarizing and concluding, critical evaluation of the topic, and quality of handouts (content, layout, and referencing). Each item is rated on a scale of 0 to 10. All seminars (50) during the study period were rated by at least two faculty members immediately after the seminar, with the raters being blind to each other’s rating. The mean of the two raters’ scores was taken. This measure eliminated the chance of rater bias. The trainees rated themselves on the same tool within 24 hours of the presentation. Informed consent from the respondents and approval of the department chair was obtained before the study. Analysis was done with SPSS Version 10, and paired t-tests were used to compare the ratings. Trainee self-assessments were higher on all 12 items than were faculty assessments (Table 1). Significant differences were shown on the following seven items: relevance of content (p50.015), organization of slides (p50.002), timemanagement (p50.009), creativity in presentation (p50.008), handout content (p50.004), quality of handouts (p50.003), and referencing (p50.017). This study attempted to compare trainee self-evaluation and faculty evaluation of resident-run seminars by use of a standardized rubric. The results revealed that trainees tended to overestimate their performance in all areas of seminar presentation. This finding is consistent with those
Academic Psychiatry, 36:6, November-December 2012
http://ap.psychiatryonline.org
Kausik Goswami, M.D., Prabha Chandra, M.D. Geetha Desai, M.D., Kandavel Thennarasu, M.D. Santosh K. Chaturvedi, M.D.
S
elf-evaluation has gained importance in medical education because it helps trainees in self-reflection and in developing insight into their strengths and weaknesses, two necessary skills for professional life (1). But do trainees really understand their weaknesses or deficiencies? Interestingly, research in self-assessment skills has shown that deficits in the ability to assess one’s own competence are common among subjects who perform poorly (2, 3). This lack of insight about one’s deficits has been ascribed to relatively poor metacognitive skills, leading to lack of understanding about weaknesses and poor training in selfassessment. This could be overcome by providing both earlier knowledge and subsequent feedback regarding the desired competency (3, 4). Over- or under-estimation of performance was observed among residents in family medicine evaluating their patient-interviewing skills (4) and pharmacy students’ selfevaluating seminar performance (5). There is limited research in this area, particularly in psychiatric residency training. Resident-run seminars or lectures are a useful and commonly-used academic exercise in several settings where the trainee has to make a presentation on a predetermined topic that is followed by a discussion involving the audience, consisting of faculty and other residents. It involves skills of compilation of literature, critical evaluation, and communication with peers and teachers. Despite the wide use of such seminars and the growing importance of self-evaluation in training, only one study, conducted among trainees in a pharmacy course, has examined this aspect (5). In this study, we compared faculty evaluation of seminar presentations with trainee self-evaluation of their own presentations. This method was chosen to examine trainees’ ability to self-rate, as self-assessment maps onto important competencies for any physician. The resident-run
493
SELF-EVALUATION BY INDIAN PSYCHIATRY TRAINEES TABLE 1. Comparison of Students’ Self-Evaluation Scores and Faculty Evaluation Scores for 50 Students Seminar Items
Faculty Evaluation
Student Self-Evaluation
t
p
7.01 (0.88) 6.90 (1.14) 6.72 (1.09) 6.85 (1.06) 6.11 (1.09) 5.88 (1.03) 5.74 (1.05) 6.37 (1.14)
7.15 (1.64) 7.44 (1.08) 7.48 (1.18) 7.47 (1.26) 6.49 (1.29) 6.51 (1.27) 6.17 (1.60) 6.68 (1.37)
0.65 2.51 3.29 2.71 1.68 2.79 1.75 1.33
NS 0.015* 0.002** 0.009** 0.100 0.008** 0.087 NS
6.10 (1.11) 6.40 (1.12) 6.40 (1.14) 6.42 (1.13) 76.56 (10.50)
6.57 (1.76) 7.10 (1.40) 7.18 (1.42) 7.09 (1.75) 83.51 (13.13)
1.62 2.98 3.08 2.48 3.13
0.111 0.004** 0.003** 0.017* 0.003**
Completeness Relevance of content Organization of slides Time-management Presentation style Creativity in presentation Handling questions Summarizing and concluding Critical evaluation Handout content Handout layout Handout referencing Total Values are t-test mean (standard deviation). * p ,0.05; ** p ,0.01.
of other studies comparing student and peer assessment in academic exercises (5–7). Several factors might contribute to this discrepancy. Students may not be clear on what is expected of them. It may also indicate inadequate selfassessment skills, resulting in an overestimation of one’s performance. Hodges et al. (4) have emphasized the need for training in self-assessment at all stages of medical education to overcome this problem. Several methods can be used to enhance self-assessment. Better and more detailed training in various aspects of seminar preparation and self-assessment might help. Benchmark presentations that help in role-modeling and practice sessions that include self-assessment are other methods. The value of immediate and analytical feedback in enhancing trainee self-assessment has also been emphasized (2, 4). Strengths of our study include an adequate sample size and use of a rating instrument with good psychometric properties. Rater bias was eliminated by independent rating and using the mean of two ratings. However, as the study was conducted only at one training site and involved only one method of training, the results may not be generalizable. Further studies need to be done using the same tool at different training sites. More studies need to be conducted on methods of enhancing self-assessment among psychiatry
494
http://ap.psychiatryonline.org
trainees. Research is also needed to examine whether enhancing self-assessment and providing effective feedback actually improve performance.
References 1. Brown S, Rust C, Gibbs G: Involving students in the assessment process, in Strategies for Diversifying Assessments in Higher Education. Oxford, Oxford Centre for Staff Development, 1994; http://www.lgu.ac.uk/deliberations/ocsd-pubs/div-ass5.html 2. Kruger J, Dunning D: Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol 1999; 77:1121–1134 3. Dunning D, Johnson K, Ehrlinger J, et al: Why People Fail to Recognize Their Own Incompetence. CDPS 2003; 12:83–87 4. Hodges B, Regehr G, Martin D: Difficulties in recognizing one’s own incompetence: novice physicians who are unskilled and unaware of it. Acad Med 2001; 76(Suppl):S87–S89 5. Maclaughlin EJ, Fike DS, Alvarez CA, et al: Reliability of a seminar grading-rubric in a Grand Rounds course. Journal of Multidisciplinary Healthcare 2010; 3:169–179 6. Stefani LAJ: Peer-, self-, and tutor-assessment: relative reliabilities. Stud High Educ 1994; 19:69 7. Albanese M, Dottl S, Mejicano G, et al: Distorted perceptions of competence and incompetence are more than regression effects. Adv Health Sci Educ Theory Pract 2006; 11:267–278
Academic Psychiatry, 36:6, November-December 2012