Comparing and contrasting undergraduate competence in musculoskeletal medicine with cardiovascular medicine and neurology
S. Basu,
C. Roberts,
D. I. Newble and
M. L. Snaith1
Academic Department of Medical Education, University of Sheffield and 1 Department of Rheumatology, Royal Hallamshire Hospital, Sheffield, UK.
Correspondence to: S. Basu. E-mail: Bazz82517{at}aol.co.uk
 |
Abstract
|
---|
Objectives. With an increasing prevalence of musculoskeletal conditions within the UK, specialty bodies are concerned that graduating medical students may lack appropriate knowledge in this system. We investigated the knowledge base of final year Sheffield medical students in the musculoskeletal system, compared with other major body systems.
Methods. A computer-based assessment was designed covering core topics that a pre-registration house officer should know about in musculoskeletal medicine, cardiology and neurology, using a predesigned testing format. The test was blueprinted against internal and external guidelines. It comprised 24 extended matching questions, each with three stems. A sample of 74 volunteer students from the final year (year 5) of the medical course at the University of Sheffield took part in the assessment.
Results. Overall scores of students on the test ranged from a baseline of 45% to a maximum of 85%. Test reliability was 0.75 (Cronbach's alpha). On stratifying the overall percentages into marks for individual systems, it was found that there were no significant differences between scores in musculoskeletal medicine, cardiovascular medicine or neurology.
Conclusions. Despite the disparity of teaching between musculoskeletal medicine and other major organ systems within Sheffield's integrated medical curriculum, the knowledge base of medical students in the basic and clinical musculoskeletal sciences appears to be similar to that for cardiovascular medicine and neurology by the time of graduation. Nevertheless, several important issues must be addressed before these findings can be generalized.
KEY WORDS: Great Britain, Musculoskeletal system, Assessment, Medical student
 |
Musculoskeletal medicine in the curriculum
|
---|
Longer life expectancy and an increasing elderly population have contributed to the growing worldwide impact of musculoskeletal conditions [1, 2]. It is estimated that the prevalence of these disorders may rise by at least 34% in Canada and 88% in Australia between 1985 and 2020 [3]. Hence, the effects of musculoskeletal disease are anticipated to place a greater financial strain upon the economies of nations across the world. With demands on service delivery likely to increase, the musculoskeletal community is concerned that a relatively small amount of specialty-specific teaching within the medical undergraduate curriculum does not reflect the system's importance [4, 5]. Consequently, medical students may lack competence in this field at the time of graduation in comparison with other body systems [6, 7]. This fear has resulted in a worldwide movement by the International League of Associations for Rheumatology to convince medical faculties of the importance of musculoskeletal education. Proposals to address the issues include at least 6 months of musculoskeletal teaching in all medical undergraduate curricula [8]. However, there appears to be little evidence within the literature to substantiate whether such concerns are justified. We therefore aimed to investigate the knowledge base of students at the University of Sheffield in the musculoskeletal system compared with other major body systems.
 |
The Sheffield undergraduate musculoskeletal course
|
---|
At the time of this study (December 2003), final year Sheffield medical students had experienced a fairly conventional curriculum. It consisted of a preclinical component largely focused on biomedical sciences, followed by a clinical component predominantly undertaken in teaching hospital settings. Basic musculoskeletal sciences were taught in the first year during a 6-week module before the Christmas vacation. In year 3, there was a timetabled 2-week clinical rotation combining orthopaedics and rheumatology. In year 4, accident and emergency medicine and trauma management were covered in a further 2-week period. Some clinical musculoskeletal medicine exposure occurred in placements such as general practice and paediatrics, but was likely to be variable between placements. In contrast, students receive between 15 and 20 weeks of cardiology and neurology training during the course depending on clinical placement rotations.
 |
Knowledge base in the musculoskeletal system
|
---|
In an earlier study [9], we investigated the knowledge levels of final year Sheffield medical students in the musculoskeletal system using a computer-based assessment. A group of senior musculoskeletal undergraduate teachers (consultants and a specialist registrar) determined a passing score for the test using a modified Angoff procedure. All participants scored above the predicted standard of 64%, indicating that these students possessed sufficient musculoskeletal knowledge for their roles as pre-registration house officers. Despite this important finding, we had still to address the concern that graduating medical students may possess lower standards of knowledge in the musculoskeletal system than in other major systems.
 |
Assessment of multiple disciplines
|
---|
Instances in which a single assessment has been used to evaluate student performance across multiple medical disciplines are scarcely documented within the literature. Student knowledge profiles in twenty-eight medical disciplines were successfully compared from one administration of a progress test in two Dutch medical schools [10]. We extended this concept to determine whether a short, feasible and more carefully constructed assessment could allow the statistical cross-comparison of the knowledge base of Sheffield students in musculoskeletal medicine, cardiology and neurology.
 |
Aims and objectives
|
---|
The aim of this study was to determine whether student knowledge levels in musculoskeletal medicine are comparable to those in cardiovascular medicine and neurology by the time of qualification. Accordingly, our objectives were: (i) to produce a computer-based competence test in musculoskeletal medicine, cardiology and neurology; (ii) to administer this test to a cohort of final year medical students at the University of Sheffield via the Internet; and (iii) to contrast the knowledge levels of final year medical students in the musculoskeletal system with those in the cardiovascular system and neurology.
 |
Methods
|
---|
Test development and implementation
A computer-based assessment was developed covering the core topics that a pre-registration house officer should know about in musculoskeletal medicine, cardiovascular medicine and neurology. It comprised 24 extended matching questions, each with three stems (72 questions in total). Since our goal was to provide an approximation of group performance across disciplines, eight extended matching questions (24 questions in total) were created to broadly sample each body system. This ensured equal weighting of subject matter. The researchers constructed each question using recommended published guidelines [11]. In the absence of any formal undergraduate musculoskeletal curriculum within the UK, important topics were identified from three sources, each of which provides suggestions for course content: (i) guidelines for the undergraduate musculoskeletal curriculum (EULAR) [12]; (ii) recommendations on undergraduate musculoskeletal training (British Orthopaedic Society, British Society of Rheumatology) [13]; and (iii) a consensus study performed at the University of Sheffield involving local musculoskeletal practitioners from the South Yorkshire region [14].
Despite their limitations as stand-alone documents, we believed that, through a process of triangulation [15, 16], the combination of these sources would provide the basis for creating a valid, reliable and feasible computer-based assessment in musculoskeletal medicine. Initially, musculoskeletal themes common to each source were identified and collated through the triangulation process. To ensure representative sampling of curriculum content, selected themes were transferred into a blueprinting framework [17] and placed under appropriate performance headings, which are displayed in Fig. 1. These categories covered the basic (anatomy, physiology, biochemistry, pathology) and clinical (history, examination, investigation, management) science competencies of musculoskeletal disease [13]. The EULAR guidelines and consensus study were then used to identify individual topics within each theme. These topics were then weighted according to importance using ratings from the consensus study and placed into corresponding sections of the blueprint. Subject matter included degenerative and inflammatory arthritis, crystal arthropathy, connective tissue disease and fractures.
In the absence of formal national guidelines, subject matter in cardiovascular medicine was identified from topics included within the core Sheffield undergraduate medical curriculum [18]. This document outlines the cardiovascular material that a newly qualified doctor from Sheffield medical school would be expected to know about. Selected cardiovascular topics were placed into the blueprint under appropriate performance headings to ensure representative sampling of curriculum content. Subject matter included arrhythmias, heart failure, physiology of conduction disorders and drug management.
Themes in basic and clinical neurology were extracted from curricular guidelines published by the British Neurological Society [19]. This document delineates the outcome objectives of the undergraduate medical curriculum in this system. Topics were placed into the blueprint under appropriate performance headings to ensure accurate weighting of curriculum material. Subject matter included neuromuscular disorders, headache, visual field defects and therapeutical side-effects. The content validity of this test was assured by accurately sampling the major learning outcomes of the Sheffield undergraduate medical course in each system. This detailed blueprinting process provided a foundation for the cross-comparison of test scores between disciplines.
At the time of this study, many of the 195 final year students were on clinical attachments distant from Sheffield and thus could not be assembled as a cohort for testing. Seventy-four volunteers completed the assessment in a prebooked computer facility at the University of Sheffield. This was done between the hours of 12 p.m. and 1 p.m. over a 3-day period during December 2003. Additionally, participants needed a unique password to access the assessment, meaning substitutes could not take part. The researcher invigilated each student to prevent cheating. Although no incentives were offered, the assessment was advertised as a revision aid before the final MB ChB written examinations to encourage participation. No students received any specialty-specific teaching in any of the three systems in the year prior to the assessment. Results were stored anonymously within a database to which only the researchers had access. On completion of the test, students were automatically given feedback for wrong answers. Prior to the study, all students provided informed consent; formal ethical approval was not required. Data were processed in SPSS Version 11.0 and Sigmaplot Version 8.0.
 |
Results
|
---|
Student performance
The overall test scores of students ranged from a baseline of 45% to a maximum score of 85%. The mean score of participants was 68.7%, with a standard deviation of 9.05. To meet our objectives, we then stratified the overall marks of the students into percentage scores for questions based upon musculoskeletal medicine, cardiovascular medicine and neurology respectively. Figure 2 displays the mean percentage scores of the assessed final year students in these medical systems, with 95% confidence intervals.

View larger version (16K):
[in this window]
[in a new window]
|
FIG. 2. Mean test scores of students knowledge, with 95% confidence intervals, in three major body systems.
|
|
A one-way analysis of variance (ANOVA) was used to look for differences between the performances of students in the different body systems covered within this assessment. We found that there were no significant differences at the 0.05 confidence level between performances in cardiology, neurology and musculoskeletal medicine (F = 0.364).
Reliability and feasibility
The reliability of this test was Cronbach's alpha = 0.75. All students who took part in the study successfully completed the assessment within the allocated 1-h time frame. No students reported any difficulties with accessing or using the computer-based administration system.
Cross-comparison of scores
We calculated the item difficulty (p value) of questions in each system to provide further evidence beyond the blueprinting process for their similarity in standard, thus allowing the cross-comparison of scores between disciplines. This is calculated by dividing the number of correct responses for each question by the total number of responses. Values range between 0 and 1, a high value indicating more correct responses. The average item difficulty of a musculoskeletal question was p = 0.71, for a cardiovascular question it was p = 0.71 and for a neurological question it was p = 0.69. Using one-way ANOVA, it was found that there were no significant differences between the difficulty indices of questions testing each system (F = 0.568).
 |
Discussion
|
---|
This computer-administered assessment provided useful information regarding the performance of final year Sheffield medical students in musculoskeletal medicine, cardiovascular medicine and neurology. Although the number of questions included within the assessment may not have been sufficient to cover the entire domain of knowledge within each system in great detail; the reliability figure (Cronbach's alpha = 0.75) would indicate that the length of our test was suitable for its intended purposeobtaining group data to compare interdisciplinary performance.
The main limitation of this study was the use of volunteers. Ideally, a random sample would have been preferable. Had we been able to access the scores of our volunteer students in their summative assessments, we could have established how well they represented the abilities of their colleagues. However, access to such information at the University of Sheffield is privileged. It could be suggested that an unrepresentative or homogeneous sample could have invalidated our results and overestimated test reliability. Nevertheless, nearly 40% of final year students (74 individuals out of a total of 195) at Sheffield medical school completed the assessment. This certainly increases the likelihood of our results being representative of the year group as a whole. Indeed, we have shown previously that group data regarding students knowledge base in musculoskeletal medicine can be obtained from samples of this size [9]. Furthermore, the overall test scores of students followed an approximately normal distribution (mean = 68.7%, median = 70.8%, mode = 70.8%) and the range of scores varied from a minimum of 45% to a maximum of 85%. Both of these suggest, at the very least, that candidates with a wide range of abilities were involved. From the questions set, it would appear that by the time of qualification, students possess a knowledge base in musculoskeletal medicine that is comparable to that in other major systems in which similar concerns regarding undergraduate proficiency have not been raised. Such findings are reassuring, adding further evidence for our belief that graduating students from Sheffield possess sufficient musculoskeletal knowledge for the role of a pre-registration house officer [9].
Study implications
This study shows that, even with a relatively short test, the knowledge base possessed by undergraduate students across different medical systems can be effectively gauged. Our findings would appear to allay faculty concerns at Sheffield that medical students require more curriculum instruction in the underpinning sciences of basic and clinical musculoskeletal medicine. Our results suggest that, as a whole, the undergraduate course is successful in providing Sheffield medical students with a knowledge base in musculoskeletal medicine that is equivalent to that in other disciplines in which more time is given to curriculum instruction. We have shown the value of careful, meticulous question construction in developing this test. Nevertheless, it is likely that concern will persist regarding the performance of students in this system until similar research using longer assessments, covering a greater range of subject matter, are conducted on a wider scale.
Generalizability of findings
Our results cannot yet be generalized beyond Sheffield medical school. Our methodology, however, provides a basis from which to develop more extensive knowledge tests in musculoskeletal medicine, and represents one way in which our findings may be reproduced. Nevertheless, until several major problems are addressed, we believe that concern regarding the knowledge base of newly qualified doctors in this system will remain. Firstly, there must be common agreement upon the core content of the undergraduate musculoskeletal curriculum. Specialty bodies may initially wish to redefine appropriate outcomes in the undergraduate musculoskeletal course, and think about relevant teaching and learning strategies to achieve these. This may be accomplished through a consensus approach [13] or the refinement of existing guidelines [12, 13, 20]. Secondly, relevant assessment frameworks that are matched to the outcome objectives of the undergraduate musculoskeletal curriculum must be designed, implemented and evaluated [17]. This is fundamental if we are to develop accurate assessments of knowledge, clinical skills and professional behaviours. Finally, robust standard-setting methods must be employed to ensure the proficiency of medical graduates in this field [21]. This is the only way we can ensure that medical students will possess sufficient competence in the musculoskeletal system for their role as pre-registration house officers.
Areas for further research
It remains to be seen whether an adequate knowledge base in the musculoskeletal system is translated into effective clinical skills [22]. This may be investigated using composite written and practical assessment formats [23]. Pilot work at the University of Sheffield suggests that students are capable of converting their knowledge base in musculoskeletal medicine into efficient clinical skills [24]. Assessing the performance of students who are interacting with patients on the wards or in the community can determine whether knowledge is put into efficient day-to-day practice. This, and the assessment of professional behaviours, may be achieved through applications such as the portfolio or video recording [25, 26]. Further research is needed to provide the evidence base for best practice in training and assessing medical students in the musculoskeletal system.
 |
Acknowledgments
|
---|
We thank the Arthritis Research Campaign for the funding of SB during his intercalated summer scholarship in Medical Education, which was supervised by CR. We thank all of the volunteer final year medical students at the University of Sheffield who participated in this study, and in particular Dr Amy Thomas for her continual assistance.
M. Snaith is a co-holder of the Arthritis Research Campaign Educational Grant. The other authors have declared no conflicts of interest.
 |
References
|
---|
- Badley EM, Webster GK, Rasooly I. The impact of musculoskeletal disorders in the population: are they just aches and pains? Findings from the 1990 Ontario Health Survey. Journal of Rheumatology 1995;22:7339.[ISI][Medline]
- Woolf AD, Akeeson K. Understanding the burden of musculoskeletal conditions. BMJ 2001;322:107980.[Free Full Text]
- Badley EM, Crotty M. An international comparison of the estimated effect of the aging of the population on the major cause of disablement, musculoskeletal disorders. Journal of Rheumatology 1995;22:193440.[ISI][Medline]
- McManus IC. The changing clinical experience of British medical students. Lancet 1993;34:9414.
- Kay L, Deighton CM, Walker D, Hay EM. Undergraduate rheumatology teaching in the UK: a survey of current practice and changes since 1990. Journal of Rheumatology 2000;39:8003.[CrossRef]
- Di Paola M, Bennet GC, Shearer JR. Orthopaedic teaching in United Kingdom medical schools. Med Teacher 1986;8:1558.[ISI]
- Freedman KB, Bernstein J. The adequacy of medical school education in musculoskeletal medicine. J Bone Joint Surg Am 1998;80:14217.[Abstract/Free Full Text]
- World Health Organization. The global economic and healthcare burden of musculoskeletal disease. WHO and the Bone and Joint Decade. Geneva: WHO, 2001. http://www.ota.org/downloads
- Basu S, Roberts C, Newble D, Snaith M. Competence in the musculoskeletal system: assessing the progression of knowledge through the Sheffield medical undergraduate course. Med Educ 2004. In press.
- Verhoeven BM, Verwijnen GM, Scherpbier AJJA, Holdrinet RSG, Oeseburg B, Bulte JA. An analysis of progress test results of PBL and non-PBL students. Med Teach 1998;20:3106.[CrossRef][ISI]
- Case SM, Swanson DB. Constructing written test question for the basic and clinical sciences. Philadelphia: National Board of Medical Examiners, 1996.
- Doherty M, Woolf AD. Guidelines for rheumatology undergraduate core curriculum. Ann Rheum Dis 1999;58:1335.[Free Full Text]
- British Orthopaedic Society, British Society of Rheumatology. Musculoskeletal curriculum for the undergraduate. London: General Medical Council, 2001;18.
- Moore DJ. What level of competence should the newly qualified doctor have in the musculoskeletal system? PhD Thesis, University of Sheffield, 2001.
- Basu S. Competence in the musculoskeletal system: assessing the progression of knowledge of students in the musculoskeletal system through the medical undergraduate course. PhD Thesis, University of Sheffield, 2002.
- Denzin N, Lincoln Y. Handbook of qualitative research. Thousand Oaks (CA): Sage, 1994.
- Newble DI, Cannon R. A handbook for medical teachers. 4th edn. Bodmin: MPG Books, 2001.
- Cardiovascular System Core Competencies. Phase 4 Course Handbook. Sheffield: Sheffield Medical School, University of Sheffield, 2004.
- Teaching neurology in the 21st century: suggestions for UK medical schools planning their core curriculum. Association of British Neurologists, 1994.
- Coady D, Walker D, Kay L. Regional examination of the musculoskeletal system (REMS): a core set of clinical skills for medical students. Rheumatology 2004;43:6339.[Abstract/Free Full Text]
- Norcini JJ. Research on standards for professional licensure and certification examinations. Eval Health Professions 1994;17:16077.[ISI]
- Simon SR, Volkan K, Hamann C, Duffey C, Fletcher SW. The relationship between second-year medical students OSCE scores and USMLE Step 1 scores. Med Teach 2002;24:5359.[CrossRef][ISI][Medline]
- Verhoeven BH, Hamers JGHC, Scherpbier AJJA, Hoogenboom R, van der Vleuten C. The effect on reliability of adding a separate written assessment component to an objective structured clinical examination. Med Educ 2000;34:5259.[CrossRef][ISI][Medline]
- Basu S. Assessing the reliability and feasibility of a computer-based and practical objective structured clinical examination in musculoskeletal medicine. PhD Thesis, University of Sheffield, 2003.
- Driessen E, van Tartwijk J, Vermunt JD, van der Vleuten CP. Use of portfolios in early undergraduate medical training. Med Teach 2003;25:1823.[CrossRef][ISI][Medline]
Submitted 5 February 2004;
Accepted 29 June 2004