Health Economics Research Group, Brunel University, Uxbridge
Department of Psychiatry, Warneford Hospital, University of Oxford, Oxford, UK
Correspondence: Teresa Jones, Health Economics Research Group, Brunel University, Uxbridge, Middlesex UB8 3PH, UK. E-mail: teresa.jones{at}brunel.ac.uk
Declaration of interest T. B. is on the editorial board of the British Journal of Psychiatry and Psychological Medicine. The National Health Service Executive, London, funded the study.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Aims To assess the perceived importance of different mental health journals to psychiatrists clinical practice and compare this with impact factors.
Method Random samples of psychiatrists providing child and adolescent, adults of working age and old age services chose up to ten journals read or consulted with regard to their clinical work, ranking the top three. For these journals, comparisons were made with impact factors and importance as outlets for UK psychiatry research.
Results A total of 560 questionnaires were completed (47%). Two membership journals (the British Journal of Psychiatry and the BMJ) were most read and highest ranked. Associations between impact factors, clinicians ratings and importance as outlets for psychiatry papers varied.
Conclusions The results could lead to reconsideration of the importance of some journals. Academic assessments of the status of journals should not be assumed to reflect their influence on clinicians.
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
![]() |
METHOD |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
National Health Service (NHS) Research Outputs Database
The Research Outputs Database was constructed by The Wellcome Trust
(Dawson et al, 1998)
and then maintained by the Centre for Information Behaviour and the Evaluation
of Research, City University. It covers the full range of research
publications, including basic and clinical sciences, in the peer-reviewed
journals contained in the Science Citation Index and Social Science Citation
Index databases. The Research Outputs Database contains bibliographic
information from biomedical papers with a UK address, including details of
funding acknowledgements. The NHS Research Outputs Database, a subset of the
Research Outputs Database, has been constructed and contains details of papers
from England that involve some element of NHS financial support
(Wellcome Trust & NHS Executive,
2001).
The NHS papers have been identified using a filter for England that identifies them via one or more of the following: characteristics of the name of the authors institution, for example hospital; the institutions postcode; or the funding acknowledgements on the paper.
Journal impact factors
Journal impact factors were obtained from the 2001 edition of the on-line
Journal Citation Reports from the Institute for Scientific Information. The
journal impact factor is a measure of the frequency with which the
"average article" in a journal has been cited in a particular year
or period. The annual JCR impact of a journal... is calculated by dividing the
number of current year citations to the source items published in that journal
during the previous two years
(Garfield, 1994). A ranking
order of journals within the psychiatry category based on
journal impact factors was also taken from the 2001 Journal Citation Reports
for each of the citation indices (i.e. the Science Citation Index and the
Social Science Citation Index).
Questionnaire survey
Psychiatrists names and addresses
A sample of 1200 registered Members and Fellows of the Royal College of
Psychiatrists was provided by the College. The Royal College of Psychiatrists
does not allow access to its membership lists but agreed to supply address
labels for one-off use to distribute the questionnaires. The NHS consultant
psychiatrists were selected randomly via the Statistical Package for the
Social Sciences statistical software (SPSS version 10.1) from within three
patient-age groups (child, adult and old age).
Psychiatrists providing services for these three patient-age groups were present in the Colleges membership list in the approximate proportions 1:3:1, but equal numbers from each group were included in the sample. This was to ensure sufficient numbers in each group to allow detailed analysis.
Selected UK psychiatrists were asked, by questionnaire survey, which journals they read or consulted on a regular basis with regard to their clinical practice. To ensure anonymity no record of the participants was kept.
Questionnaire structure
A list of journals containing psychiatric research was extracted from the
NHS Research Outputs Database using a mental health filter previously
developed for the Research Outputs Database
(Wellcome Trust & NHS Executive,
2001). The journals were ordered according to the number of papers
on psychiatric research they published. To limit the list used in the study,
the top 32 journals, accounting for 60% of UK psychiatry papers in the period
19901999, were presented in alphabetical order on the questionnaire.
The questionnaire recipients were asked to tick up to ten journals that they
read or consulted on a regular basis with regard to their clinical work and to
rank the top three of these. In doing this they were invited to add journals
missing from the list that they considered important for mental health
clinical practice. They were then asked to provide brief details of the type
of NHS contract they held, the number of academic and clinical sessions they
worked, which patient age-group they worked with and which disorders they
covered. The questionnaire is appended as a data supplement to the on-line
version of this paper and is available from the authors on request.
Questionnaire analysis
The data from the returned questionnaires were entered into a database.
There were difficulties in identifying all the journals added to the
questionnaires by the respondents, particularly because of the similarity of
some journal names; therefore, a member of the Health Economics Research Group
at Brunel University (Avril Cook, see Acknowledgements) independently checked
these and the journal names were verified using Ulrichs
International Periodicals Directory
(Bowker, 2003) or the
internet.
The psychiatrists responses were collated and tabulated according to their type of practice, and the relationships between their rankings, journal readership and impact factors were examined.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
A substantial number of recipients added more journals to the questionnaire, bringing the total number of journals from the original 32 up to 156. Those journals that were not listed on the original questionnaire but appear in the summary tables have been marked with an asterisk. The 560 respondents ticked or added journal names on 3215 occasions. Out of these, 13 (0.4%) related to 10 unverified journal names that nevertheless were included within the database.
The difference in the median number of journals read by psychiatrists with
academic commitments (ten journals or more) and those without (three journals)
was statistically significant (KruskalWallis;
=7.823, P=0.005). The percentage
of non-academic psychiatrists reading three journals or fewer
was higher across all three age groupings but especially so in the adult group
where the figure was 40%. Overall, approximately twice as many academics
compared with non-academics read at least ten journals.
Tables 1 and 2 detail the specific journals that psychiatrists read and have ranked first, second or third with regard to their clinical work. A striking consistency was found at the top of each table, both across all age groups and between academic and non-academic psychiatrists. The two most prominent journals across the board were the British Journal of Psychiatry followed by the BMJ.
|
|
Table 1 shows that for adult psychiatrists these two journals dominated their reading habits. Both were cited by over 90% of the sample whereas the third most commonly cited journal (the American Journal of Psychiatry) was read by only 50%. Although both the British Journal of Psychiatry and the BMJ were still cited by about 90% of the child and old age psychiatrists, specialised journals became more prominent.
The importance of these specialised journals for child and old age psychiatrists was even more clearly reflected in their ranking of the journals (Table 2). Within each category there were only a small number of journals (between four and seven) ranked in the top three in importance for their clinical practice by more than 10% of psychiatrists (i.e. only a few are widely viewed as important; see Table 2).
Relationship of readership to the NHS Research Outputs Database and journal impact factors
Table 3 contains the same 31
journals as those that appear in Table
1: those journals read by 10% or more of psychiatrists in any
category (by patient age or academic commitments). The numbers of psychiatry
papers in the journals between 1990 and 1999 were identified through the NHS
Research Outputs Database. The British Journal of Psychiatry is
clearly the journal with the largest number of papers in the psychiatry
section of the NHS Research Outputs Database and also is ranked as the most
important to clinical practice overall. The pattern needs to be interpreted
with caution because only a relatively small proportion of BMJ papers
are related to psychiatry and the Archives of General Psychiatry
publishes comparatively few papers per year.
|
The data on journal impact factors are presented in a number of ways in Table 3, including (in the final column) their position in the Institute for Scientific Information ranking of psychiatry journals, which does not include general medical journals such as the BMJ or the Lancet.
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Few key journals
For respondents, it appears that a small number of journals are very
important for the dissemination of information with a bearing on clinical
practice. Furthermore, the numbers of psychiatrists who read three journals or
fewer are most marked in the adult group, which is approximately three times
as large as the other two categories in the membership of the Royal College of
Psychiatrists. Adjustment for this factor would give overall figures for
psychiatrists reading three journals or fewer of 27%, with 11% for academics
and 34% for non-academics. If a small number of journals are of greatest
importance to clinicians, then research findings published in these journals
have greater potential to result in benefit to patients.
The journals that were found to be the most important to clinical psychiatrists the British Journal of Psychiatry and the BMJ are both available as part of membership to the Royal College of Psychiatrists and the British Medical Association, respectively. Most psychiatrists, therefore, will receive them without subscription. Because our sample was provided by the Royal College of Psychiatrists from lists of its Members and Fellows, then all will receive the British Journal of Psychiatry. Approximately 80% of practising doctors are members of the British Medical Association and therefore automatically will receive the BMJ. In addition, the BMJ is freely accessible via the internet. These two journals were noticeably ahead of all the other journals in terms of readership. The British Journal of Psychiatry also contained a significant proportion of the total papers in the whole NHS Research Outputs Database mental health field. It published more than three times as many articles funded in some way by the NHS as the International Journal of Geriatric Psychiatry in second place. Rafferty et al (2000) found that one dominant journal in the field of nursing contained far more publications than any other (46% of the total), followed by a second containing 6.5%.
Comparisons with journal impact factors
The journal impact factor has been used as an indicator of the quality of
research published within journals
(Schwartz & Lopez Hellin,
1996). However, our study shows that the correlation between the
perceived importance attributed by clinical practitioners in the field of
psychiatry and the journal impact factor is neither a simple nor a consistent
relationship (see Fig. 1). This
was found for clinicians both with and without academic commitments.
|
The journal impact factor scores of the British Journal of Psychiatry and the BMJ are reasonably high. The British Journal of Psychiatry was positioned eighth out of 81 within the field of psychiatry, and the BMJ, although obviously not listed in the psychiatry journals, had a journal impact factor that would have put it in third position. Of the top ten journals in the field of psychiatry, according to journal impact factors, only four (Archives of General Psychiatry, Biological Psychiatry, the American Journal of Psychiatry and the British Journal of Psychiatry) were found in the 31 journals with a clinical readership of 10% or more in any one category (by patient age-group or academic commitments) and the first three of these were found to be of significantly greater importance to academics in the adult group than to any other category of psychiatrist.
In this discussion an inevitable limitation has to be considered. As discussed later, national bias has been found in both publication trends and readership of journals (Grant et al, 2000; Schein et al, 2000), therefore it was felt that a standard list of journals that were most important as outlets for UK psychiatry publications would seem an appropriate starting point for this survey. Including a comparatively small number of such journals enabled a manageable list to be included in the questionnaire, but allowing clinicians to add journal names inevitably created two populations of journals. It would seem reasonable to assume that those journals included within the questionnaire were more likely to be ticked as read than those not included. The Archives of General Psychiatry, which has the highest journal impact factor in the field of psychiatry, was not one of the journals listed on the questionnaire owing to the small number of UK papers published in it. This absence from the questionnaire might have reduced the numbers of psychiatrists referring to the Archives of General Psychiatry but would not account for the marked differences found between academics and non-academics and between the adult group compared with the other two patient age categories. It is possible of course that papers in journals such as this have an important indirect, rather than direct, influence on clinical practice in the UK through their impact on guidelines, etc.
The Journal of Child Psychology and Psychiatry and Allied Disciplines is widely read and the most highly rated by child psychiatrists, but its journal impact factor would put it in 17th position if it were included in the Science Citation Index listing for psychiatry in the Journal Citation Reports. The International Journal of Geriatric Psychiatry is similarly widely read within its patient age-group of psychiatrists but is poorly rated by journal impact factor, being positioned 38th in the psychiatry list. Clinical Child Psychology and Psychiatry and Advances in Psychiatric Treatment feature quite prominently in the results of this survey but neither was included in the original questionnaire because neither is listed by the Science Citation Index.
Survey findings in context
Previous research has raised several relevant issues, including the
significance of country of publication and the relevance of journal impact
factor to readership patterns. Grant et al
(2000) examined UK clinical
guidelines to determine the flow of information from basic research to
clinical practice and the nationality of papers cited in UK clinical
guidelines. They found that UK authors of clinical guidelines cite UK
publications in a greater proportion (25%) than is found in world biomedical
literature (10%). Schein et al surveyed 1000 Fellows of the American
College of Surgeons and found that they were only interested in American
journals, despite the fact that an international survey by e-mail had found a
UK journal, the British Journal of Surgery, to be the
best general surgical journal in the world
(Schein et al, 2000).
Furthermore, journal impact factor was not a consideration for these surgeons
when selecting journals to read. Lewison
(2002) examined the
relationship between the importance of journals to researchers and to the
users of research in a series of medical sub-fields. He found significant
variations. In the more clinical subjects such as nursing there was virtually
no correlation between their perceived relative importance and the citation
score of the journal.
The findings of this survey indicate that in terms of nationality of journals read, and ranked first, second or third, although some of the American journals are of considerable significance there is a clear bias towards journals published in the UK. This bias may, however, have been exaggerated by the choice of journals listed in the original questionnaire.
The study examines what psychiatrists read and perceive as important to their clinical practice, and it covers a large number of publications. Adopting a broad approach in a brief questionnaire inevitably means that some issues were not explored. The term read has not been examined, just as the different sections and article types in the journals have not been analysed individually. Further studies, possibly on a journal-by-journal basis, would provide more information in this area (Tyrer, 2003). Also, there may have been some differences in the respondents interpretation of the questions asked, which possibly will have had some effect on the findings. A further limitation that has not been examined here involves the variation in journal availability to psychiatrists, with expensive ones available only to some psychiatrists. Research on these issues would provide additional information on the factors involved in translating biomedical research into clinical practice.
By itself, transmission through journals is not seen as a major way of securing the implementation of research findings (Coomarasamy et al, 2001) but clinicians do believe that journals are their main source of information (Schein et al, 2000). Assessment of the impact of journals is an important part of a wider stream of work being developed to examine the value of research (Buxton & Hanney, 1996) and, in turn, such analysis (Hanney et al, 2003) is being linked to work on how best to implement research findings (Grimshaw et al, 2001). Perhaps greater recognition should be given to researchers who publish in the journals that are of greatest importance in disseminating research, irrespective of their journal impact factors.
![]() |
Clinical Implications and Limitations |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Limitations
![]() |
ACKNOWLEDGMENTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Buxton, M. & Hanney, S. (1996) How can payback from health services research be assessed? Journal of Health Services Research & Policy, 1, 35-43.[Medline]
Coomarasamy, A., Gee, H., Publicover, M., et al (2001) Medical journals and effective dissemination of health research. Health Information and Libraries Journal, 18, 183 -191.[CrossRef][Medline]
Dawson, G., Lucocq, B., Cottrell, R., et al (1998) Mapping the Landscape: National Biomedical Research Outputs 1988-95. London: Wellcome Trust.
Garfield, E. (1994) The impact factor. Current Contents, 20 June. Available at http://www.isinet.com/essays/journalcitationreports/7.html/
Grant, J., Cottrell, R., Cluzeau, F., et al
(2000) Evaluating payback on biomedical
research from papers cited in clinical guidelines: applied bibliometric study.
BMJ, 320, 1107
-1111.
Grimshaw, J., Shirran, L., Thomas, R., et al (2001) Changing provider behaviour. An overview of systematic reviews of interventions. Medical Care, 39 (suppl. 2), 112 -1145.
Hanney, S., Soper, B. & Buxton, M. (2003) Evaluation of the NHS R&D Implementation Methods Programme. Uxbridge: Brunel University.
Lewison, G. (2002) Researchers and users perceptions of the relative standing of biomedical papers in different journals. Scientometrics, 53, 229 -240.[CrossRef]
Lewison, G., Rafferty, A. & Traynor, M. (2001) Is nursing research typical of biomedical research? Research Evaluation, 10, 97 -103.
Medical Research Council (2002) MRC Annual Report and Accounts 01/02. London: Medical Research Council.
Rafferty, A., Traynor, M. & Lewison, G. (2000) Measuring the Outputs of Nursing R&D. A Third Working Paper. London: Centre for Policy in Nursing Research.
Schein, M., Paladugu, R., Sutija, V., et al (2000) What American surgeons read: a survey of a thousand Fellows of the American College of Surgeons. Current Surgery, 57, 252 -258.[CrossRef][Medline]
Schwartz, S. & Lopez Hellin, J. (1996) Measuring the impact of scientific publications. The case of the biomedical sciences. Scientometrics, 35, 119 -132.
Tomlinson, S. (2000) The research assessment
exercise and medical research. BMJ,
320, 636
-639.
Tyrer, P. (2003) A new dawn for the yellow
journal? Editors response (letter). British Journal of
Psychiatry, 183, 459
-465.
Wellcome Trust & NHS Executive (2001) Putting NHS Research on the Map. An Analysis of Scientific Publications in England, 1990-97. London: Wellcome Trust and NHS Executive.
Received for publication November 18, 2003. Revision received March 24, 2004. Accepted for publication April 5, 2004.
Related articles in BJP: