INTER-RATER RELIABILITY OF THE ASSESSMENT OF COMMUNICATION SKILLS IN PHARMACOLOGY

Shankar P R * , Das B * , Partha P * , Shenoy N * , Dubey A K * Effective communication skills are required for medical students and doctors. Communication skills training and assessment in Pharmacology is conducted in English using simulated patients. The assessment procedure has not been tested for inter-rater reliability. To overcome this deficiency, the present study was carried out to obtain information on the agreement between two independent observers simultaneously evaluating the students using a standardized check list. The study was conducted on 175 second and fourth semester medical students at the Manipal College of Medical Sciences, Pokhara, during the month of December 2001. Two observers, who had agreed beforehand on the scoring criteria and the checklist to be used, simultaneously and independently evaluated the students. Kappa test was used to measure agreement. There was very good to moderate agreement regarding the total scores. Very good to moderate agreement was also seen for the parameters: drug-related factors and non-drug factors. Individual factors showed poor agreement because of the ‘kappa paradox’. The findings, overall, demonstrate the validity of the system of assessment. This was a preliminary study and more detailed studies are required before modifications in the system of communication skills assessment can be suggested.


INTRODUCTION
Effective communication skills are necessary to take an accurate medical history, improve patient compliance and satisfaction with treatment.Proper communication with the patient and their relatives optimizes clinical outcome, provides emotional comfort and assures informed consent of the patients.It is important therefore, for medical students to develop a good standard of communication skills.The development of these skills during the undergraduate training has been a concern of educators. 1Studies have shown deterioration in these skills as students' progress through medical school.shown that introducing CST in the pre-clinical years did indeed lead to higher levels of skills but patient contact was important. 4ese days the short-case examination is being gradually replaced by the objective structured clinical examination (OSCE) which substitutes real patients with standardized simulations. 5In communication skills in Pharmacology we use simulated patients and concentrate on the drug-related aspects of the case.The training and assessment are carried out in English and the time available is 3 minutes.
In the Kathmandu University, the subjects of Anatomy, Physiology, Biochemistry, Pathology, Microbiology and Pharmacology are taught during the first 4 semesters of the MBBS course.Problems in Pharmacotherapeutics will serve as an appropriate introduction to students of the concept of communication skills.
At the Department of Pharmacology we discuss the important points which the students as future doctors have to communicate regarding the treatment of different diseases.However, this is just an introduction to the concept of communication skills and has a number of drawbacks.The students are given a preformed diagnosis and do not have to elicit information from the patient to arrive at the same.The time allotted is only 3 minutes and it may not be possible to give all relevant information to the standardized patient.
There has been great interest in the psychometric properties of assessment data obtained by means of OSCE/OSPE (Objective structured practical examination). 6,7Previous studies in residents and clinical students have shown OSCE/OSPE to be a reliable and valid form of examination. 8,9One should be careful, however, in extrapolating these results to undergraduate medical students.Our system of communication skills assessment (CSA) is carried out at a single station, usually by a single examiner using a checklist.The points looked for are broadly the same for the different problems.Though CSA has been carried out in our institute for over 5 years the system has not been tested for inter-rater reliability.
The objectives of the present study was to obtain information on the agreement between two independent observers simultaneously evaluating the students using a standardized check list in the system of assessment of communication skills.

METHODS
The The two observers assessed the student over a 3 minute assessment period using the checklist.Different batches of students were assessed on 7 consecutive days using a different clinical problem on each day.Each individual observer was unaware of the scores of the other observer till the exercise was over and the data were tabulated for analysis.
Calculating kappa values assessed agreement among the raters regarding the individual grading points, the broad categories of points and the overall score.A kappa value of 0.81-1 indicated very good agreement, a value between 0.61-0.8indicated substantial agreement and between 0.41-0.6indicated moderate agreement.Values between 0.21-0.4indicated fair agreement while values less than 0.2 indicated poor agreement.The mean, standard deviation (SD) and range of the scores given by the two observers were calculated for the following parameters: total score, 'drug-related' factors and 'definition of the problem'.
The simulated patients given to the students were suffering from epilepsy (day 1), arthritis (day 2), leprosy (day 3), bronchial asthma (day 4), amoebiasis (day 5), hypertension associated with bronchial asthma (day 6) and peptic ulcer (day 7).The students had to define the problem in layman's language and give proper drug-related and non-drug information to the patient.

RESULTS
The agreement for the 'total score' between the two observers is shown in table I.As regards the 'total scores' there was very good to moderate agreement between the two observers.The kappa values for the parameter 'definition of the problem' are as follows: epilepsy (?=0.31), arthritis (?=0.64), leprosy (? =0.25), bronchial asthma (? =0.39), amoebiasis (? =0.2), hypertension associated with bronchial asthma (?=0.55) and peptic ulcer (?=0.39).There was a good agreement between the two raters as regards assessment of the parameter 'definition of the problem'.But the number of scoring categories was low (0, 0.25, 0.5) and most of the students had similar scores.In this situation the scores were not able to give information on the agreement between the two observers.This is known as 'kappa paradox'.Since most of the individual assessment points were scored as 0, 0.25 and 0.5 because of the kappa paradox we are not reporting the individual category scores.

Communication skill subject
The kappa values for the 'drug factors' group is shown in table II.There was again very good to moderate agreement between the two observers.The kappa values for the 'nonpharmacological measures' are shown in table III.Substantial agreement was observed for some of the diseases while fair or slight agreement was observed for other diseases.
The mean, SD and range was calculated for the parameters 'total score', 'drug-related' factors and 'definition of the problem'.For the 'total score', the mean, SD and range for the 1 st observer were 3.65, 1.76 and 4 while for the 2 nd observer the values were 3.77, 1.82 and 4. For the parameter 'drug-related' factors the mean, SD and range for the 1 st observer were 1.56, 0.36 and 1.75 while for the 2 nd observer the corresponding values were 1.66, 0.48 and 1.75.For the parameter 'definition of the problem' the mean, SD and range for the 1 st & 2 nd observers were 0.38, 0.15, 0.5 and 0.41, 0.16 and 0.5 respectively.

DISCUSSION
There was very good to moderate agreement between the two observers as regards the 'total scores', the 'drug-related' factors and the 'communication' factors.This finding is supportive of the robustness of the OSPE format (used for communication skills assessment) to threats from validity other than the performance being tested. 10,11e obvious advantage of the OSPE/OSCE method is that the outcome of a communication skills course can be assessed easily and quickly.Though the overall agreement between the observers was good, the individual scoring points showed a low kappa value due to the already explained kappa paradox.Way to reduce this paradox as suggested by our statistician may be increasing the number of scoring categories for each of the points being tested.However, this will make the scoring more difficult and also more time will be required for the assessment.
At present communication skills assessment is being conducted along with the spotter component of the practical examination in Pharmacology.In the spotter component of the examination there are 10 spots.The student gets three minutes at each of the spots.The student has to choose the appropriate drug for the given disease condition and answer one or two questions related to the chosen drug.The surrogate patients used do not assess the student performance.Assessment by surrogate patients has been shown to produce a different score with respect to trained observers. 7 checklist is used to assess the skills and this has been widely used to standardize assessment of these skills. 12major drawback is that there is no further development of CST in our institution.For the training to be really effective it has to be incorporated throughout the continuum of undergraduate medical education.The students should be introduced to communication skills during the semesters devoted to basic science teaching, the concept should be developed during the clinical postings and reinforced during the internship period.
Traditionally most clinicians rely on an informal learning of communication skills by the students, during ward rounds and in the out-patient department.Even in Europe very little formal teaching time is provided to communication skills development. 13CST is helpful to effective practice and there should be increasing emphasis on this aspect of medical education.The revised curriculum of Kathmandu University has introduced CST in the course of Pharmacology teaching. 14e CST is being conducted in English and not the vernacular language.The students as future doctors will have to communicate with the majority of the patients in different vernacular languages.
Given the diverse background of our students English seems to be the only common language of communication.We agree with Gispert et al. 11 that the assessment procedures need to be periodically reviewed to confirm that testing remains unbiased.Our system of assessment showed good inter-observer agreement as regards the total scores.In the examinations the maximum marks for the communication skills are 5. Communication skills account for 10% of the total marks in the Pharmacology practical examination.Communication skills are ultimately linked to clinical ability.A physician who establishes rapport quickly will be more successful in taking an efficient and thorough history and ensuring better patient compliance and satisfaction with treatment.

CONCLUSIONS
In our study, very good to moderate agreement was seen between the two observers.The findings, overall demonstrate the validity of the system of communication skills assessment.This was a preliminary study and more detailed studies with an increased number of observers are required before we can suggest modifications in the system of communication skills assessment.The use of English, as the language of communication skills training and assessment was necessary, as we have a diverse students from different national, linguistic and socioeconomic backgrounds.With medical schools in Nepal, admitting students from diverse backgrounds, the issue of language to be used for communication skills training and assessment may become increasingly important.

2 , 3
One of the reasons cited was the introduction of communication skills training midway through the Bachelor of Medicine and Bachelor of Surgery (MBBS) course.At the Manipal College of Medical Sciences, affiliated to the Address for correspondence : Dr. P. Ravi Shankar Manipal College of Medical Sciences P.O.Box: 155, Deep Heights, Pokhara, Nepal.Email: pathiyilravi@rediffmail.com, pathiyilravi@hotmail.com Kathmandu University we follow an integrated curriculum with early introduction of students to patients.Keeping in mind the importance of communication skills, the Department of Pharmacology decided to introduce communication skills training (CST) right from the first semester.Evans et al. have study was carried out on 175 second and fourth semester medical students during the month of December 2001 at the Manipal College of Medical Sciences, Pokhara.The scoring criteria and the checklist were agreed upon beforehand by the two observers.The individual parameters were broadly grouped into the following categories: a) definition of the problem b) non pharmacological measures c) drug factors (correct choice of drug, correct dose, frequency and duration of administration, description of adverse effects, drug interactions) and d) communication factors (follow up, feedback, allowing the patient to ask questions and the overall style of communication).

Table I : Degree of agreement between the two observers regarding the total scores
* A kappa value of 0.81-1 indicates very good agreement, a value between 0.61-0.8indicates substantial agreement and 0.41-0.6indicates moderate agreement.Values between 0.21-0.4indicate fair agreement, values less than 0.2 indicates poor agreement.