Inter-rater reliability of the assessment of communication skills in Pharmacology

Abstract

Effective communication skills are required for medical students and doctors. Communication skills training
and assessment in Pharmacology is conducted in English using simulated patients. The assessment procedure
has not been tested for inter-rater reliability. To overcome this deficiency, the present study was carried out
to obtain information on the agreement between two independent observers simultaneously evaluating the
students using a standardized check list. The study was conducted on 175 second and fourth semester
medical students at the Manipal College of Medical Sciences, Pokhara, during the month of December
2001. Two observers, who had agreed beforehand on the scoring criteria and the checklist to be used,
simultaneously and independently evaluated the students. Kappa test was used to measure agreement. There
was very good to moderate agreement regarding the total scores. Very good to moderate agreement was also
seen for the parameters: drug-related factors and non-drug factors. Individual factors showed poor agreement
because of the ‘kappa paradox’. The findings, overall, demonstrate the validity of the system of assessment.
This was a preliminary study and more detailed studies are required before modifications in the system of
communication skills assessment can be suggested.
Key Words: Communication, Educational measurement, Undergraduate medical education.