360-degree assessment in a multidisciplinary team setting

T. B. Potter and R. G. Palmer

Birmingham Heartlands and Solihull Hospital NHS Trust (Teaching), Solihull, UK.

Correspondence to: R. G. Palmer, Clinical Director of Rheumatology, Birmingham Heartlands and Solihull Hospital NHS Trust (Teaching), Lode Lane, Solihull, West Midlands B91 2JL, UK. E-mail: Robert.Palmer{at}heartsol.wmids.nhs.uk


    Abstract
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflict of interest
 References
 
Objectives. To use the 360-degree assessment in the multidisciplinary setting of a rheumatology department and to evaluate its impact, recognizing that this process will become part of the revalidation process of NHS professionals in the future.

Methods. Seventeen team members completed an anonymous questionnaire to give confidential opinions about the clinical, humanistic and other skills of their colleagues. Results and comments were collated and given as feedback to each individual. Before feedback, participants were asked to predict their perceived strengths and weaknesses. After feedback they evaluated the process.

Results. A profile of abilities was established for each team member and discussed privately with the clinical director. Often team members had good insight into their perceived strengths and weaknesses. Some participants were hurt by negative comments made about them even if this was balanced by positive comments. There were mixed views on the relevance and usefulness of the process, and whether or not it should be repeated. Some team members found the process threatening.

Conclusion. The 360-degree assessment can be used in a multidisciplinary setting, the questions being the same for all individuals. It is a very powerful tool that must be handled carefully so that it does not cause more harm than good.

KEY WORDS: 360-degree assessment, Multidisciplinary team


    Introduction
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflict of interest
 References
 
There is an increasing emphasis on evaluating performance for all medical personnel and allied health professionals within the NHS. This will be given greater prominence by the proposals from the General Medical Council that the consultant revalidation procedure will include some form of peer assessment, and this is likely to be the 360-degree assessment. As competence is multifactorial, there is a clear need to assess characteristics that define a good practitioner—not only medical knowledge but also humanistic skills. The 360-degree assessment is becoming particularly important in industry, where senior and junior colleagues and customers can rate a worker's performance [1]. It involves an identical questionnaire being completed by at least 10 colleagues, collated and then fed back to the individual.

There is developing interest in this method within medicine, where it has also been shown that peer ratings are reliable if at least 10 or 11 raters are used [2]. Interestingly, the outcome is not affected by the person assessed choosing his or her own raters; the relationship between the rater and the subject does not affect the results [3]. A recent pilot study has suggested that 360-degree assessment of senior house officers was a practical and acceptable method for gathering information on humanistic qualities. Participants were able to choose their own raters and the majority found the process fair and helpful [4].

We report our experience of using the 360-degree assessment in a multidisciplinary team setting to inform the annual appraisal process that occurs in a rheumatology directorate in a teaching hospital. Ethics approval was not required.


    Methods
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflict of interest
 References
 
All 17 team members participated in the assessment. There were four consultants, two middle grade doctors, three nurse specialists, one physiotherapist, two occupational therapists and five medical secretaries. Each individual was based at one of the two hospital sites. A questionnaire with explanatory notes was distributed to each via the hospital intranet, which consisted of 10 statements and a space for free comments. The statements were scored on a Likert scale of 1–5 (1 = totally disagree, 5 = fully agree). The statements, adapted from published work [4], were as follows:

Gets on very well with patients;

Gets on very well with staff;

Is always attentive to the needs of others in the team;

Gives a high-quality clinical opinion;

Always responds in a timely manner;

Is always well organized;

Dictates high-quality clinical letters;

Works well even under pressure;

This person is very good at his/her job;

Always inspires others and is appropriately ambitious.

The same statements were considered applicable to every member of staff, except that for the secretaries the respondents were asked to ignore the statements relating to clinical opinion and dictation of letters. Participants had the choice of completing the form in print or by hand, returning it anonymously to the clinical director using the hospital mail; the intranet was not used for returning responses because anonymity would not have been maintained. Self-scoring was discouraged with a warning that outlying results would be ignored. The e-mails were distributed in three phases over a 6-week period to prevent overload and to increase compliance: first the doctors, followed by the allied health professionals, and then the secretaries.

The clinical director collated the results and comments and fed them back to each participant in a one-to-one interview that lasted up to 25 min. In an effort to make this a less threatening process, the clinical director's own results, both positive and negative, were discussed first. An individual was not aware of the results of others. Each person was given a histogram of his/her own results together with the mean results for his/her own professional group. Comments from colleagues were discussed. Prior to the feedback the participants were asked to predict their two highest and two lowest scores to give a measure of insight into how others perceived them.

After completion, the process was evaluated by the use of a questionnaire to seek the participants’ views on the assessment: Was it relevant? Was it useful? Was it threatening, Should it be repeated?


    Results
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflict of interest
 References
 
After the first questionnaire had been sent out, one medical and two non-medical team members wrote individually to the clinical director expressing reservations about the process and indicating that time constraints would prevent them from participating. It is not known if they subsequently completed an anonymous response. Nevertheless, all other team members gave views about their colleagues, including views about those who did not wish to be involved. Each questionnaire took less than 5 min to complete.

It was not possible for some members of staff to respond to all 10 questions about other individuals, especially if they worked on the other hospital site (e.g. a secretary from one site might not know how well a doctor from the other site related to patients). There were 10–14 responses for doctors, 9–11 responses for nurses, occupational therapists and the physiotherapist, and 6–9 responses for the medical secretaries. Each individual received between two and 11 comments. There was no relationship between the number and attitude of comments and an individual's mean score for each question. The lowest mean score for any single question was 2.6 and the highest was 5. Most scores were between 3 and 5. An example of typical results that could apply to a colleague with good clinical skills but poor communication skills is shown in Fig. 1.



View larger version (29K):
[in this window]
[in a new window]
 
FIG. 1. Example of results that may apply to a colleague who has good clinical skills and poor communication skills.

 
At the start of the feedback, each team member was asked to predict which statements would be the two highest-scoring and which would be the two lowest-scoring. For the 16 participants receiving feedback, the maximum total number of correct predictions would be 32 for highest scores and 32 for lowest scores. Results showed that 23 of the highest and 14 of the lowest scores were predicted. This could suggest that individuals have some insight into what others may say about them and that they may have greater insight into their strengths than their weaknesses.

After the feedback, all 17 team members completed a brief evaluation questionnaire. The findings are shown in Fig. 2. There was an almost even split between positive and negative responses to the suggestion that the assessment was relevant and useful, and should be repeated. The majority of participants did not find it threatening, although a small group did feel threatened by the process.



View larger version (43K):
[in this window]
[in a new window]
 
FIG. 2. Evaluation results.

 
Several learning points were gained both from the experience of the introduction of the assessment and from the comments of participants in the evaluation.

The process was time-consuming for the organizer, particularly the analysis of results.

A comment was made that the anonymous nature of the assessment led people to make more personal comments than were appropriate. Indeed, for some individuals a single negative comment was hurtful even if it was balanced by several positive comments.

The potential difficulty of using the same questions for all team members irrespective of discipline was raised.


    Discussion
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflict of interest
 References
 
The 360-degree assessment is becoming a useful tool for the assessment of humanistic skills and is becoming important within medicine. This study shows that it can be applied to a multidisciplinary team with some success and therefore could be used not only in rheumatology units but also in other speciality teams. The results can be used in the future for consultants to include in their portfolio for General Medical Council revalidation. A potential weakness of the study is that there is no easy measure of the validity of the questions, although the validity of the technique itself has been established [5]. However, the high success rate of self-prediction of scores suggests that the results are meaningful. It is of interest that individuals may have greater insight into their strengths than into their weaknesses.

The assessment was undertaken in three stages to make it more acceptable to staff, starting with the doctors. It was hoped that this would also reduce the chance of it being threatening to non-medical staff. In addition, it was hoped that the staging would help maximize cooperation. Nevertheless, the numbers of responses declined with each stage, which could reflect diminishing enthusiasm for involvement.

The evaluation revealed divided opinions on the process. Three team members had already expressed concerns about the assessment before it started and some felt threatened by the process, gaining little from it. This may simply have been due to lack of familiarity with the new process, and with time and repeated assessments the concern may diminish. In a pilot study involving junior doctors, most of the participants found the process to be helpful [4]. It is possible that some NHS personnel are more able to adapt to a new assessment process than others. Direct involvement of all staff in the development of the process, possibly with the input of professional line managers, may make the 360-degree assessment more acceptable.

Some participants found it hurtful to receive anonymous negative comments. Clearly, this assessment is a very powerful tool that needs to be handled with care so that it does not cause more harm than good. There was no relationship between the overall scores received and the number of comments or the nature of the comments, whether positive or negative. This suggests that respondents were trying to be fair, finding good things to say about the weaker team members whilst being willing to be critical of the stronger team members.

There are several important learning points. It might be preferable for future assessments to be organized by an independent person who is not a member of the team, and for the completed forms to be scannable, to reduce time and effort. Another possibility would be for feedback to be given by an individual's line manager at the time of appraisal rather than by the clinical director. The overall feeling is that the 360-degree assessment has been reasonably successful and that its role will become more important in a multidisciplinary team setting in the future.


    Conflict of interest
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflict of interest
 References
 
There was no funding for this study and no conflict of interest for the authors or participants.


    Acknowledgments
 
We thank all members of the rheumatology team for participating in this assessment.


    References
 Top
 Abstract
 Introduction
 Methods
 Results
 Discussion
 Conflict of interest
 References
 

  1. Bracken DW, Timmreck CW, Fleenor JW, Summers I. 360 feedback from another angle. Hum Res Manage 2001;40:3–20.[CrossRef][ISI]
  2. Ramsey PG, Carline JD, Blank LL et al. Feasibility of hospital based use of peer ratings to evaluate the performances of practicing physicians. Acad Med 1996;71:364–70.[Abstract]
  3. Ramsey PG, Weirich MD, Carline JD et al. Use of peer ratings to evaluate physician performance. J Am Med Assoc 1993;269:1655–60.[Abstract]
  4. Whitehouse A, Walzman M, Wall D. Pilot study of 360 degree assessment of personal skills to inform record of in training assessments for senior house officers. Hosp Med 2002;63:1–5.
  5. Church A. Do higher performing managers actually receive better ratings? A validation of multirater assessment methodology. Consult Psychol J Practice Res 2000;52:99–116.[CrossRef]
Submitted 29 April 2002; Accepted 25 June 2002